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FOREWORD 


The  U.S.  Army  Research  Institute  is  examining  the  use  of  distance  learning  technologies 
for  use  by  soldiers  in  an  “on  demand”  environment,  where  training  becomes  more  soldier 
centered  rather  than  classroom  based.  An  early  part  of  the  work  involves  a  thorough  assessment 
of  the  distance  learning  research  literature  as  it  pertains  to  training. 

A  goal  of  the  TRAINTODAY  project  is  to  determine  whether  it  is  better  to  train  a 
particular  task  up-front  in  the  classroom  or  just-in-time  in  the  field.  Since  the  application  of 
distance  learning  is  fundamental  to  this  project,  an  important  first  step  was  to  understand  the 
learning  that  it  mediates.  A  thorough  assessment  of  the  literature  in  distance  learning, 
specifically  as  it  applies  to  training  rather  than  education,  was  required.  The  results  of  this 
assessment  were  briefed  to  BG  Jon  Root  and  members  of  the  U.S.  Army  Reserve  Distance 
Learning  Futures  Group  on  23  January  1999.  These  findings  will  influence  the  incorporation  of 
stronger  research  designs  into  the  TRAINTODAY  project. 


TA  M.  SMUTIS 
'echnical  Director 


TRAINING  THROUGH  DISTANCE  LEARNING:  AN  ASSESSMENT  OF  RESEARCH 
FINDINGS 

EXECUTIVE  SUMMARY _ _ _ _ _ 

Research  Requirement: 

The  Army  has  an  ambitious  plan  to  convert  over  500  courses  to  a  distance  learning 
format,  delivering  training  to  a  soldier  when  needed  —  in  unit  learning  centers,  at  the  job  site,  or 
in  their  residence.  In  anticipation  of  this  changing  training  environment,  the  U.S.  Army 
Research  Institute  initiated  the  TRAINTODAY  project,  the  goal  of  which  is  to  determine 
whether  it  is  better  to  train  up-front  in  the  classroom  or  just-in-time  in  the  field.  Since  the 
application  of  distance  learning  is  fundamental  to  this  research,  a  first  step  was  to  understand  the 
learning  that  it  mediates.  A  review  of  the  literature  in  distance  learning,  specifically  as  it  applies 
to  training,  was  required.  In  particular,  an  assessment  of  the  experimental  designs,  reporting,  and 
interpretability  of  the  findings  were  needed  as  previous  reviews  have  focused  on  whether  there 
was  “no  significant  difference”  between  distance  learning  and  classroom  comparison  groups. 

Procedure: 

Searches  on  relevant  research  databases  resulted  in  2,000  entries,  the  large  majority 
related  to  education.  Papers  presented  at  conferences  over  the  past  three  years  were  reviewed, 
and  the  Internet  was  queried  for  studies  with  documented  experimental  designs.  The  selection 
process  resulted  in  the  codification  of  43  research  reports  that  were  most  relevant  to  the  research. 

Findings: 

Generally  speaking,  the  distance  learning  research  literature  focused  on  education  rather 
than  training,  was  largely  anecdotal,  and  when  effectiveness  was  examined,  it  was  not  supported 
by  strong  experimental  or  quasi -experimental  design.  The  studies  analyzed  in  this  report,  with  a 
combined  sample  size  of  5,438  students,  painted  a  generally  positive  view  of  the  effectiveness  of 
distance  learning  for  training  applications.  However,  for  an  understanding  of  why  distance 
learning  might  be  effective,  there  were  problems  with  reporting  some  key  instructional 
components:  40%  of  the  reports  did  not  mention  course  conversion  or  redesign,  25%  did  not 
mention  instructional  techniques,  and  50%  of  reports  did  not  fully  describe  the  method  of 
communication  (technology)  employed.  Video  teletraining  was  the  dominant  technology 
reported.  Comparison  groups  were  available  only  one-third  of  the  time.  Based  on  the  reported 
literature,  when  distance  learning  was  shown  to  be  effective,  it  was  difficult  to  determine  why. 

Utilization  of  Findings: 

These  findings  will  influence  the  incorporation  of  stronger  research  designs  into  the 
TRAINTODAY  project.  It  should  serve  as  a  reminder  to  the  research  community  to  be  more 
thorough  in  reporting  findings.  There  was  negligible  coverage  of  knowledge  retention  and 
comparisons  between  up-front  or  just-in-time  training  options,  critical  constructs  for  the 
TRAINTODAY  project. 
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INTRODUCTION 


The  Army  plans  to  convert  over  500  courses  to  a  distance  learning  format,  delivering 
training  to  a  soldier  when  needed.  As  described  in  the  Total  Army  Distance  Learning  Plan,  in  the 
21st  Century,  soldiers  will  attend  streamlined  resident  courses,  preparing  themselves  through 
diagnostic-driven,  self-paced  distance  learning  modules  delivered  at  home  station  in  unit  learning 
centers,  at  the  job  site,  or  in  their  residence.  In  recent  years,  there  has  been  an  escalating  interest 
on  the  part  of  academia,  industry,  as  well  as  the  military  to  shift  from  the  traditional  face-to-face 
classroom  to  a  distance  learning  format.  This  shift  has  been  driven  by  an  interest  in  reducing 
training  costs  while  increasing  accessibility. 

Much  of  the  interest  in  distance  learning  relies  on  the  “no  significant  difference” 
phenomenon,  which  argues  that  distance  learning  is  as  effective  as  classroom  instruction.  The 
present  report  offers  a  critical  examination  of  the  research  literature  on  the  effectiveness  of 
distance  learning,  especially  as  it  relates  to  training  rather  than  education.  This  report  is  intended 
for  training  analysts,  researchers,  and  others  needing  a  more  thorough  understanding  of  that 
literature. 

In  a  variety  of  forms,  distance  learning  has  been  an  alternative  to  the  traditional 
classroom  for  over  one  hundred  years.  The  term  distance  learning,  also  termed  distance 
education  (Femstudium  in  German),  and  most  recently  distributed  learning,  was  coined  by  Otto 
Peters  and  other  practitioners  at  the  University  of  Tubigen  in  the  1960s  (Moore  &  Kearsley, 

1996).  Research  on  distance  learning  (DL)  began  appearing  approximately  fifty  years  ago,  with 
monographs  and  articles  on  the  topic  becoming  common  in  the  1960s  (Holmberg,  1987). 

Several  defining  characteristic  of  DL  are:  teachers  and  learners  separated  physically  while 
instruction  occurs  synchronically,  the  presence  of  noncontiguous  communication  between 
student  and  teacher  (through  electronic  media  or  print),  and  the  volitional  control  of  learning  by 
the  student  rather  than  the  instructor  (Keegan,  1986;  Sherry,  1996). 

One  definition  of  DL,  articulated  by  the  Los  Alamos  National  Laboratory,  is  the 
“structured  learning  that  takes  place  without  the  physical  presence  of  the  instructor.”  This 
definition  has  been  accepted  by  the  U.S.  Distance  Learning  Association  and  by  military, 
government,  education,  and  private  sector  activities  concerned  with  the  development  and  use  of 
DL.  With  the  recent  emphasis  on  the  application  of  electronic  technology  to  DL,  an  extended 
definition  has  also  been  suggested: 
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“Distance  education  is  planned  learning  that  normally  occurs  in  a 
different  place  from  teaching  and  as  a  result  requires  special  techniques 

« 

for  course  design,  special  instructional  techniques,  special  methods  of 
communication  by  electronic  and  other  technology,  as  well  as  special 
organizational  and  administrative  arrangements.”  (Moore  &  Kearsley, 

1996,  p.  2) 

This  report  reviews  the  research  on  the  effects  of  these  techniques,  particularly 
instructional  techniques  and  methods  of  communication  on  learning.  Other  topics  of  interest 
include  interactions  between  student  and  instructor,  individual  differences,  and  the  cost- 
effectiveness  of  DL  programs. 

The  available  documentation  on  DL  is  vast  and  grows  weekly.  Several  thousand  journal 
articles,  book  chapters,  research  reports,  and  other  forms  of  technical  documentation  have  been 
published  on  the  topic.  Remarkably,  a  recent  analysis  by  the  Air  Force  indicated  that  there  were 
over  1,000,000  DL  hits  from  an  Internet  search  (Vombrock,  1998).  In  addition  to  the  many 
Internet  postings,  there  are  several  scholarly  journals  that  focus  specifically  on  DL.  The  key 
refereed  research  journals  in  the  field  are  The  American  Journal  of  Distance  Education,  the 
Journal  of  Distance  Education,  Research  in  Distance  Education,  Distance  Education,  and  Open 
Learning.  Research  articles  also  appear  in  numerous  journals  related  to  educational  technology, 
instruction,  education,  and  psychology. 

This  report  offers  a  review  of  the  literature  that,  while  certainly  not  exhaustive,  is 
representative  of  its  methodologies  and  findings.  Our  goal  is  to  form  a  perspective  after 
reviewing  studies  concerning  DL  as  applied  to  training.  It  should  be  noted  that  when 
summarizing  the  previous  reviews  and  analyses  of  DL  literature:  (1)  most  research  is  anecdotal; 
(2)  most  focuses  on  education  rather  than  training;  (3)  when  effectiveness  is  examined  it  is 
usually  not  supported  by  a  strong  experimental  or  quasi-experimental  design;  (4)  when 
effectiveness  is  measured  comparative  test  results  are  reported  approximately  one-third  of  the 
time;  and  (5)  when  data  are  reported  there  are  analytic  problems  and  errors  in  reporting  that  are 
often  overlooked  by  researchers.  This  review  should  both  summarize  what  is  known  and  serve 
as  a  framework  for  future  research  on  distance  learning. 
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Our  focus  will  be  on  training  rather  than  education.  Education,  by  its  nature,  is  open- 
ended  whereas  training  is  linked  to  organizational  objectives  and  correcting  deficiencies  in  the 
workforce  (Kraiger  &  Jung,  1997).  Although  both  encompass  learning,  our  interest  is  in  what 
DL  can  contribute  to  “a  planned  effort  by  a  company  to  facilitate  employee’s  learning  of  job- 
related  competencies,”  as  training  is  defined  by  Noe  (1999,  p.3).  Unlike  DL  applications  for 
education,  which  tend  to  orient  learning  to  achievement  on  a  written  test,  DL  for  training  should 
involve  “the  systematic  acquisition  of  skills,  rules,  concepts,  or  attitudes  that  result  in  improved 
performance  in  another  environment”  as  identified  by  Goldstein  (1993,  p.3).  There  are  gray 
areas  within  this  distinction,  such  as  the  instruction  that  occurs  in  colleges  and  universities  for 
technical,  certificate  and  graduate-level  courses  within  a  particular  career  field.  On  a  selective 
basis,  such  courses  will  be  included  in  the  review. 

DISTANCE  LEARNING  TECHNOLOGIES 
Although  DL  is  often  equated  with  technology,  print  in  the  form  of  correspondence  study 
is  still  the  most  prevalent  medium  used  in  distance  learning  (Moore  &  Kearsley,  1996). 
Computer-based  training  is  another  popular  medium.  These  two  media  will  not  be  examined  in 
this  report  unless  they  are  applied  in  combination  with  other  methods  of  instructional 
communication.  Readers  may  refer  to  Hannafin,  Hannafin,  Hooper,  Rieber  &  Kine  (1996)  for  a 
review  of  computer-based  training  and  Pittman  (1990)  for  a  review  of  the  print  medium.  This 
report  focuses  on  DL  implementations  that  apply  information  technology  to  link  audio,  video, 
text,  and  graphic  images  between  two  or  more  sites  for  the  purposes  of  training.  Before 
reviewing  the  research  findings,  a  brief  survey  of  the  growth  of  DL  will  establish  the  importance 
for  such  an  examination  of  the  literature. 


Types  of  Technology 

The  majority  of  DL  classes  employ  video  teletraining  (VTT)  technologies.  Interaction 
among  the  instructors  and  students  is  provided  using  two-way  video  with  two-way  audio  (57%  of 
classes)  or  one-way  video  with  two-way  audio  (25%);  furthermore,  one-way  prerecorded  video, 
either  alone  or  in  combination  with  interactive  video,  is  used  in  52%  of  DL  classes  (National 
Center  for  Educational  Statistics,  1997;  Walsh,  Gibson,  Miller,  &  Hsieh,  1996).  Most 
organizations  offering  DL  use  a  combination  of  technologies.  Video  is  used  more  often  by 
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industry  than  academic  or  military  organizations.  Also,  computer-based  training  is  used  least  by 
academia,  whereas  industry  uses  this  type  of  technology  85%  of  the  time  (Walsh  et  al.,  1996). 

Given  the  dominance  of  VTT,  there  are  delivery  options  concerning  analog  versus  digital 
transmission,  compression  of  the  digital  signal,  satellite  versus  terrestrial,  in  addition  to 
implementation  considerations.  Terrestrial  broadcast  offers  additional  options  for  microwave, 
fiber  optic,  cable,  or  telephone  transmission.  Their  advantages  and  disadvantages  are  described 
in  Redding  and  Fletcher  (1994).  As  the  telecommunication  industry  advances,  hew  technologies 
are  being  introduced  while  others  become  outmoded.  This  change  offers  new  options  for 
education  and  training  delivery,  such  as  audio  and  video  streaming,  computer  telephony,  viewer 
response  keypads,  and  collaborative  learning,  all  of  which  are  being  energetically  pursued.  At 
the  same  time,  researchers  in  educational  technology  are  designing  innovative  learning 
environments  to  take  advantage  of  these  emerging  technologies  (Duffy,  1997;  Duffy,  Lowyck,  & 
Jonassen,  1993).  Table  1  provides  a  description  of  current  DL  delivery  methods. 

Table  1. 

Summary  of  Delivery  Methods  of  Distance  Learning 

PRINT  Delivered  through  mail,  facsimile,  or  downloaded  from  the  Internet 

Correspondence  study  Training  Manuals  Study  Guides 

AUDIO  Delivered  over  cassette  players,  personal  computer,  telephone,  radio,  or  the  Internet 

Audio  cassettes  Compact  disc  Voicemail 

Audio  conferencing  Radio  broadcast 

Audio  teletraining  Streaming  audio 

VIDEO  Delivered  over  videocassette  players,  personal  computer,  satellite,  microwave, 
fiber  optic,  cable,  telephone,  or  the  Internet 

One-way  video,  two-way  audio  CD-ROM  Streaming  video 

Two-way  video,  two-way  audio  DVD  Videocassette 

COMPUTER-MEDIATED  CONFERENCING  -  Delivered  through  computer  networks 

Application  sharing  Bulletin  board  E-mail 

Audiographics  Chat  Room  White  Board 

COMPUTER-BASED  TRAINING  -  Stand-alone  training  applications;  audio  and  video  as  above 
Includes  many  computer  peripherals,  mass  storage  devices,  printers,  etc. 
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Implementation  Practices 

Researchers  address  the  relative  effectiveness  of  DL  for  a  variety  of  tasks,  conditions, 
and  standards,  usually  in  comparison  to  a  traditional  classroom  setting.  Practitioners,  of  course, 
must  determine  the  best  course  of  action  based  on  how-to  publications,  recommendations  from 
experts,  or  a  sensible  analysis  of  their  needs,  organizational  readiness,  and  training  budgets.  The 
basis  of  many  practices  derive  from  earlier  research  and  evaluation  findings  that  report  the 
successes,  and  failures,  of  DL  experiments,  implementation  efforts,  and  documented  success 
stories. 

Two  recent  publications  that  we  recommend  to  the  practitioners  are  Chute,  Thompson 
and  Hancock  (1999)  andMantyla  and  Gividen  (1997).  These  volumes  describe  how 
practitioners  must  first  identify  the  performance  shortfall  they  seek  to  remedy  and  then 
determine  whether  DL  is  an  appropriate  alternative.  If  DL  is  viable,  will  the  organization  accept 
the  changes  to  training  schedules  and  methods?  How  do  you  match  technologies  with  training 
requirements?  How  can  one  document  the  return  on  investment?  Will  there  be  continued 
funding  for  a  DL  endeavor?  These  issues  are  addressed  in  the  aforementioned  books.  One 
example  from  Chute  et  al.  (1999)  is  the  application  of  a  decision-making  tool  for  selecting  a 
training  technology:  “IF  the  organization  uses  electronic  mail,  bulletin  boards,  and  Internet-type 
applications  AND  your  learning  requires  interaction  between  participants  THEN  consider  a  DL 
application  which  uses  network  based  CBT,  chat  rooms  or  bulletin  board  technology”  (Table  5- 
2,  p.  76).  In  view  of  the  rapid  advancements  in  technology,  practitioners  must  maintain  a 
constant  awareness  of  documented  successes  in  the  DL  field.  They  must  also  beware  of  the 
limits  of  technology  in  workplace  learning  as  suggested  by  others  (e.g.,  Gordon,  1999;  Morgan, 
Ponticell,  &  Gordon,  1998). 


TRENDS  IN  DISTANCE  LEARNING 
Distance  learning  has  grown  enormously  in  recent  years,  and  its  prospects  are 
tremendous  as  academia,  industry,  and  the  military  increase  their  investments  in  its  use.  The 
confluence  of  advances  in  information  technologies,  a  globally  competitive  economy,  and  the 
changing  needs  of  the  workplace  require  convenient  access  to  resources  for  knowledge  and  skill 
acquisition.  Advances  in  telecommunications  and  the  desire  of  educational  and  training 
institutions  to  penetrate  new  market  segments  have  rendered  distance  learning  a  common  term. 
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Technology-based  training  systems  have  grown  40%  annually  in  recent  years,  and  analysts 
expect  this  trend  to  continue.  "This  market  is  really  taking  off,  and  tremendous  attention  is  being 
paid  by  venture  capitalists  and  other  private  investors,"  says  the  editor  of  an  online  training 
newsletter  {Investor's  Business  Daily,  December  2, 1998).  An  underlying  force  of  this  growth 
has  been  a  technology  push  by  the  telecommunication  suppliers,  the  education  and  training 
industry,  and  the  military  to  make  a  determined  commitment  toward  DL  as  a  universal 
alternative  to  the  traditional  classroom. 

There  are  many  visible  signs  of  this  growth.  In  1995,  more  students  enrolled  in  DL 
courses  than  entered  all  the  U.S.  colleges  and  universities  as  freshmen.  However,  it  is  unclear  if 
enrollment  statistics  between  these  groups  are  comparable  since  DL  courses  are  often  much 
shorter  in  duration  and  do  not  have  enrollment  figures  as  accurate  as  those  of  traditional  colleges 
and  universities.  In  a  benchmarking  forum  conducted  by  the  American  Society  for  Training  and 
Development,  certain  trends  in  DL  were  identified.  These  trends,  displayed  in  Table  2,  reflect 
the  growing  presence  of  the  DL  alternative. 

Table  2. 

Percent  of  Companies  Using  Selected  Delivery  Systems,  1994-1996 


Year 

Televised  DL 

Intemet/network  DL 

1994 

47% 

12% 

1995 

64% 

33% 

1996 

69% 

3% 

DL  in  Business  and  Industry 

Training  50  million  American  workers  is  big  business  indeed,  with  over  60  billion  dollars 
allocated  to  training  in  business  and  industry  in  1998.  There  is  direct  evidence  that  investments 
in  workplace  learning  improve  a  company’s  financial  performance.  For  example,  in  a  study  that 
examined  the  relationship  between  training  expenditures  per  employee  and  net  sales  and  gross 
profit  for  40  publicly  traded  companies,  researchers  found  that  companies  with  the  larger  per- 
employee  training  cost  had  a  57%  higher  net  sales  per  employee  and  a  37%  higher  gross  profit 
per-employee  than  those  companies  with  a  relatively  smaller  per-employee  training  cost  (Bassi 
&  McMurer,  1998).  The  same  study  also  found  that  companies  with  leading-edge  training 
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practices  spent  up  to  6%  of  payroll  on  training  and  used  high-end  methods,  such  as  the  Internet, 
teleconferencing,  and  computer-based  training  to  provide  opportunities  for  workplace  learning. 

Given  the  relationship  between  training  and  profitability,  industry  is  further  motivated  to 
increase  the  cost-effectiveness  of  training  through  the  application  of  DL  technologies.  There  is 
much  evidence  that  this  practice  is  increasing.  For  example,  the  Public  Broadcast  Service  (PBS) 
has  started  “The  Business  Channel,”  offering  an  extensive  array  of  business  education  and 
training  using  satellite  and  fiber  optics  networks  with  options  for  video-on-demand  to  the 
desktop.  PBS  is  already  the  nation’s  single  largest  broker  of  college  courseware  in  the  nation, 
enrolling  over  400,000  students  in  telecourses  through  the  Going  the  Distance  partnership  with 
colleges  and  local  stations.  The  National  Technological  University  offers  degree  and  non-degree 
programs  in  engineering  and  technical  specialties  through  the  redistribution  of  satellite  signals 
from  an  originating  university  to  multiple  locations  across  the  country.  The  consortium  of  45 
leading  universities  affiliated  with  the  National  Technological  University  are  paid  by  the 
recipient  organization,  which  is  equipped  with  a  satellite  antenna,  a  classroom,  and  students  in 
need  of  updating  their  technical  skills  while  avoiding  travel  and  time  away  from  the  workplace. 

More  than  150  corporations  possess  a  business  television  capability,  which  is  simply  a 
private  satellite  television  network  capable  of  broadcasting  live  television  programs  to  multiple 
locations.  One  example  is  the  Ford  Motor  Corporation’s  WORDSTAR  program.  In  this  network 
a  small  aperture  satellite  antenna  is  installed  at  each  Ford  and  Lincoln-Mercury  dealership  across 
North  America  for  the  purpose  of  providing  training  on  product  information,  repair,  sales,  and 
service  to  over  190,000  dealership  employees  in  North  America.  Using  digital  video 
technologies,  this  satellite-based  communications  network,  when  completed,  will  be  the  largest 
privately  owned  network  in  the  world,  offering  the  highest  number  of  concurrent  events  on  over 
20  channels.  Monthly  schedules,  patterned  after  a  television  guide,  list  the  many  course 
offerings.  The  results  of  this  paradigm  change  in  training  have  been  impressive.  The  number  of 
course  enrollments  has  increased  from  just  over  150,000  in  1994  to  over  717,000  for  the  past 
year.  Training  is  now  convenient  and  effective.  For  example,  the  number  of  “false  positive” 
returns  of  component  parts  sent  back  as  faulty  has  decreased  in  direct  relationship  to  those 
employees  who  participated  in  the  maintenance  training  on  a  particular  component  (Conley, 
1998). 
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On  another  front,  Web-based  training  for  the  information  technology  workforce  will 
grow  from  $92  million  in  1996  to  $1.7  billion  in  2000,  a  growth  of  over  1,800%,  with  an 
emphasis  on  Intranet-based,  asynchronous,  self-paced  instruction  {Web  Week,  Sep,  8, 1997). 
Internet  tools  emerging  in  the  training  marketplace  to  simulate  the  traditional  classroom  and 
increase  interactions  between  instructors-and-students  as  well  as  students-and-students  include 
internet  relay  chat,  multi-user  dimensions,  and  multi-user  simulation  environments  (Kouki  & 
Wright,  1996).  Table  3  summarizes  some  of  the  recent  trends  in  distance  learning  marketplaces. 

Table  3. 

Distance  Learning  Statistics  (Phillips,  1998) 

•  Number  of  students  taking  distance  learning  courses  from  higher-education 
institutions  as  of  late  1997:  7,000,000 

•  Number  of  accredited  degree  and  certificate  distance  learning  programs:  1 ,200 

•  Number  of  accredited  distance  learning  colleges:  900 

•  Percentage  of  corporate  training  delivered  online  in  1997:  16% 

•  Percentage  of  corporate  training  estimated  to  be  delivered  online  in  2000:  28% 


DL  in  the  Military 

Training  in  the  military  is  substantial  with  a  yearly  training  load  (full-time  equivalent 
students)  of  165,000  (Military  Manpower  Training  Report,  1998).  Millions  of  dollars  are  spent 
on  travel  costs  to  transport  soldiers,  sailors,  and  airmen  to  centralized  training  facilities  for 
periods  ranging  from  several  days  to  several  months.  The  potential  for  cost  savings  through  DL 
is  clearly  substantial.  There  is  similar  interest  in  military  departments  in  Europe  as  well  (Seidel 
&  Chatelier,  1994). 

All  service  branches  are  active  in  the  implementation  of  DL  technologies  to  replace 
traditional  classrooms  for  both  active  and  reserve  components  (Metzko,  Redding,  &  Fletcher, 
1996).  The  Army,  for  example,  has  recently  approved  a  comprehensive  plan  for  the  Regular 
Army  and  the  Reserve  Component  (comprising  both  the  Army  Reserve  and  Army  National 
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Guard).  This  plan  calls  for  the  preparation  of  745  classrooms  at  home  and  abroad  as  well  as 
mobile  and  deployable  classrooms  for  specialized  training  requirements.  The  plan  calls  for  the 
conversion  of  525  courses  to  a  DL  format.  Fielding  of  the  facilities  is  being  timed  to  coincide 
with  the  availability  of  multimedia  courseware  and  the  infrastructure  required  to  deliver  VTT 
courses  and  other  training  products.  The  delivery  system  combines  fiber  optics,  asynchronous 
transfer  mode  (ATM)  switching,  and  some  satellite  transmissions  to  link  classrooms,  field 
training,  armories,  and  reserve  centers.  Implementing  the  Army’s  DL  system  is  planned  to 
continue  through  2010. 

Of  special  note  is  the  DL  initiative  of  the  Army  National  Guard.  The  Guard  maintains 
nearly  3,200  armories  in  54  states  and  territories,  all  of  which  could  be  used  as  DL  training 
centers.  Properly  configured  and  managed,  such  an  arrangement  can  overcome  the 
disadvantages  of  geographical  dispersion  of  members  and  the  inconveniences  of  an  intermittent 
training  schedule.  An  additional  feature  of  the  Army  Guard’s  network  is  shared  usage,  whereby 
other  government,  educational,  and  civic  organizations,  as  well  as  private  businesses,  can  use  the 
network  and  DL  classrooms  on  a  fee  basis  when  it  is  available  (Byrne,  1998).  Since  Guardsmen 
generally  use  the  armories  only  a  few  days  each  month,  they  can  also  serve  many  non-defense 
educational  and  training  purposes. 

The  Navy  does  not  have  a  formal  DL  plan  per  se.  Rather,  it  is  integrated  into  an  overall 
plan  for  training  technology  which  encompasses  both  active  duty  and  reserve  sailors.  Examples 
of  DL  in  the  Navy  include  an  interactive  VTT  network  that  is  used  for  both  DL  and 
teleconferencing.  The  system  uses  satellites  to  broadcast  to  ships  at  sea  and  telecommunication 
lines  to  deliver  courses  on  shore.  The  network  consists  of  19  sites  in  major  fleet  concentration 
areas  and  25  networked  classrooms.  It  is  available  24  hours  per  day  and  on  weekends  to  serve 
the  training  needs  of  the  Naval  Reserve  Force.  In  1997,  the  Navy  offered  52  courses  of 
instruction  through  its  network.  All  students  receive  the  same  certificates  of  completion  as  those 
who  attend  traditional  classes.  The  Naval  Postgraduate  School  has  initiated  a  DL  program  for 
its  students  on  the  network.  Navy  medical  centers  also  make  use  of  this  system,  both  to  conduct 
courses  and  to  practice  telemedicine.  Marines  at  Navy  locations  also  take  courses  on  the 
network. 

The  Marine  Corps  is  planning  a  wide  reaching  and  interconnected  Marine  Corps 
Learning  Network  (MarineNet)  to  support  individual  learning.  The  concept  is  dependent  on 
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previously  planned  upgrades  to  the  their  telecommunications  and  network  infrastructure  as  well 
as  the  conversion  of  traditional  courses  to  electronic  media  over  the  next  several  years.  The 
initiative  includes  growing  use  of  the  Internet  for  student  information,  registration,  and 
administration.  The  Marine  Corps  is  converting  their  skills  training  courses  to  a  DL  format, 
moving  toward  courses  written  for  the  Internet  and  Intranet  delivery.  The  expectation  is  that 
every  military  occupational  skill  in  the  Marine  Corps  will  have  a  DL  module.  Plans  also  call  for 
a  Training  and  Education  Point  of  Presence  to  be  installed  at  each  Marine  Corps  base  and  station. 
Every  networked  workstation  will  have  access  to  digital  training  materials.  The  Marine  Corps  is 
also  mindful  that  approximately  60%  of  its  training  is  provided  by  the  other  services,  and  their 
DL  network  must  be  able  to  access  those  training  resources. 

The  Air  Force  is  planning  the  development  of  a  world-class  DL  education  and  training 
system  and  to  that  end  established  the  Air  Force  Institute  for  Distance  Learning  in  1998. 
Currently,  tens  of  thousands  of  Air  Force  students  around  the  globe  take  courses  delivered 
through  a  variety  of  DL  formats.  These  numbers  are  anticipated  to  grow  in  the  future,  as 
investments  are  made  in  organization,  facilities,  course  conversions,  telecommunications 
infrastructure,  and  instruction  for  faculty  and  courseware  developers.  As  a  result  of  trends 
identified  in  the  Air  Force  2025  Project,  the  Air  Force  expects  that  education  and  training 
programs  may  be  adjusted  to  meet  the  individual  needs  of  students  in  an  anytime-anywhere 
environment. 

The  Air  Force  Institute  of  Technology  provides  graduate,  professional,  and  specialized 
education  to  members  of  the  Air  Force  and  has  a  growing  distance  learning  presence.  This 
institute  is  responsible  for  everything  but  content,  dealing  with  course  conversion, 
administration,  and  broadcasts  of  distance  learning  events  (Westfall,  1998).  Also,  the  Air  Force 
Reserve  Component  is  to  produce  and  deliver  distributed  learning  through  many  delivery 
systems,  including  the  Internet. 

The  Services  operate  eight  separately  managed  VTT  networks.  Each  network  can  send 
and  receive  courses  using  internal  network  resources,  and  can  model  commercial  broadcasting 
with  one-way  video  or  video  telecommunication  with  two-way  video.  Both  t3^es  use  two-way 
audio  to  deliver  instruction.  Table  4  identifies  the  eight  networks  used  by  the  Services. 
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Table  4. 


Department  of  Defense  YTT  Networks  (G.A.  Redding,  1998,  personal  communication) 


Network 

Video  Caoabilitv 

Audio  Caoabilitv 

Chief  of  Naval  Education  and  Training 

Electronic  Schoolhouse  Network 

Two-way 

Two-way 

Navy  Medical  Education  and  Training  Network 

One-way  &  two-way 

Two-way 

Marine  Corps  Satellite  Education  Network 

Two-way 

Two-way 

Satellite  Education  Network  -  Army 

One-way 

Two-way 

Training  Network  —  Army 

Two-way 

Two-way 

Air  Technology  Network  —  Air  Force 

One-way 

Two-way 

Air  Force  Reserve  Network 

Two-way 

Two-way 

Warrior  Network — Air  National  Guard 

One-way 

Two-way 

The  military  has  conducted  extensive  research  and  analyses  on  the  effectiveness  of 
distance  learning  (cf.  Barry  &  Runyan,  1995;  Howard,  1997).  As  the  implementation  of  DL 
continues,  research  will  likely  continue  on  topics  such  as  training  on  demand,  digital  skills,  and 
collaborative  team  training.  In  higher  education,  gaps  in  the  research  have  been  identified  in 
areas  such  as  learning  styles,  drop-out  rates,  and  interaction  of  multiple  technologies  (The 
Institute  for  Higher  Education  Policy,  1999).  There  will  be  a  continuing  interest  in  analyses  to 
determine  return  on  investment,  to  provide  baseline  measures  for  continued  improvement  of 
training  products  and  processes,  and  to  prepare  for  new  training  technologies  emerging  from 
academic,  industrial,  and  military  laboratories. 

Distributed  Learning 

The  Advanced  Distributed  Learning  (ADL)  initiative  is  a  Department  of  Defense 
undertaking  whose  strategy  is  to  promote  widespread  collaboration,  exploit  network-based 
technologies,  lower  development  costs,  and  develop  next  generation  learning  technologies  by 
creating  reusable  content  at  a  lower  cost  using  object-based  tools.  The  Advanced  Distributed 
Learning  initiative  will  employ  an  Instructional  Management  System  (IMS)  to  tag  and  manage 
the  learning  objects  produced  as  a  result  of  the  initiative.  For  example,  an  object  representing  an 
animation  of  a  moving  truck  created  for  an  Army  training  program  could  be  clipped  from  the 
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IMS  and  pasted  into  an  Air  Force  training  program  or  even  a  commercial  training  program.  One 
of  the  goals  is  to  develop  an  automated  system  that  will  perform  information  management 
functions  by  keying  on  the  IMS  indexing  feature.  The  ADL  initiative  will  provide  a  unified 
“system  of  systems”  for  use  by  all  Department  of  Defense  personnel  (Office  of  the 
Undersecretary  of  Defense  for  Personnel  and  Readiness,  1999). 

DL  in  Academia 

In  1995,  there  were  25,730  DL  courses  offered  by  institutions  of  higher  education  in  the 
United  States  with  an  estimated  690  degree  programs  and  170  certificates  offered  exclusively  at 
a  distance  (National  Center  for  Education  Statistics,  1997).  From  1996  to  1998,  distance 
education  was  projected  to  grow  dramatically  in  a  variety  of  remote  sites:  branch  campuses, 
student’s  homes,  work  sites,  libraries,  community-based  organizations,  and  correctional 
institutions.  Approximately  75%  of  the  institutions  currently  offering  distance  education  courses 
have  plans  to  begin  or  increase  their  use  of  two-way  interactive  video  and  two-way  online 
interactions.  The  1997  Campus  Computing  Project  survey  shows  that  nearly  33%  of  courses 
offered  at  the  605  institutions  polled  use  electronic  mail,  up  25%  from  1996.  The  same  survey 
indicated  that  the  percentage  of  courses  at  private  universities  using  e-mail  is  60%,  and  nearly 
half  of  public  university  courses  use  the  electronic-mail  medium  {Chronicle  of  Higher 
Education,  Oct.  17,  1997). 

Unlike  industry  and  the  military  that  focus  on  DL  as  a  cost-effective  alternative  to  the 
traditional  classroom,  academia  must  focus  on  DL  as  providing  a  value-added  commodity.  Both 
industry  and  the  military  can  justify  DL  costs  as  a  means  to  avoid  travel  costs  and  the 
productivity  loss  of  workers  in  transit  to  training.  Here,  the  claim  that  training  is  “as  good  as” 
the  traditional  classroom  is  acceptable.  Since  the  students  are  already  present  at  the  traditional 
university,  justifying  costs  for  DL  cannot  stem  from  a  reduction  in  travel  costs  or  productivity 
but  rather  focus  on  improvements  in  learning  in  which  education  and  training  is  “better  than”  the 
classroom.  Examples  of  “better  than”  include  new  methods  of  instruction,  such  as  collaboration 
with  remote  sites  and  asynchronous  conferencing.  Of  course,  in  some  cases  DL  can  offer  greater 
access  for  students  at  satellite  campuses,  in  which  case  the  learning  outcomes  do  not  necessarily 
need  to  be  better  than  the  classroom.  This  situation  is  changing  as  colleges  and  universities 
expand  their  markets  and  compete  for  students.  In  higher  education,  shortcomings  in  the 


12 


research  on  the  educational  effectiveness  in  DL  have  been  recognized;  gaps  in  the  research  have 
been  identified  in  areas  such  as  learning  styles,  drop-out  rates,  and  the  impact  of  interaction  of 
multiple  technologies  (The  Institute  for  Higher  Education  Policy,  1999). 

PREVIOUS  REVIEWS  OF  DISTANCE  LEARNING 

The  published  literature  on  DL  is  overwhelmingly  anecdotal.  Evaluations  are  usually 
informal  and  conducted  by  users  rather  than  independent  sources.  Large-scale  evaluations  have 
tended  to  focus  on  issues  such  as  usability,  learner  preferences,  and  equipment  quality  rather  than 
learner  outcome  (Council  of  Chief  State  School  Officers,  1995).  In  the  voluminous  literature, 
comparatively  little  research  has  examined  teaching  effectiveness  in  DL  (Webster  &  Hackley, 
1997)  and  most  of  this  has  been  oriented  towards  education  rather  than  training.  For  example,  of 
the  241  articles  cited  by  Russell  (1998)  demonstrating  a  “no  significant  difference”  effect 
between  traditional  classroom  instruction  and  instruction  through  distance  learning,  our  analysis 
indicates  that  only  13%  focus  on  traditional  training  courses;  much  of  this  work  was  conducted 
in  the  1950s  and  early  1960s  with  print  and  instructional  television.  It  should  be  noted  that 
failing  to  find  a  significant  difference  between  DL  and  traditional  classrooms  does  not,  from  a 
statistical  perspective,  mean  that  the  two  are  the  same.  Failing  to  reject  the  null  hypothesis  (in 
this  case  that  DL  and  traditional  classrooms  produce  the  same  outcomes)  should  not  be 
interpreted  as  evidence  that  the  null  hypothesis  is  true.  Instead  it  is  appropriate  to  suspend 
judgement  due  to  a  lack  of  sufficient  evidence  (Dayton,  1970). 

Several  researchers  have  reviewed  earlier  findings  and  examined  research  trends  on  the 
topic.  Scriven  (1991)  reviewed  109  articles  published  in  the  Distance  Education  journal  during 
its  first  ten  years  and  classified  them  by  topic.  The  three  largest  topics  were  (a)  students  and 
their  characteristics  (23  articles),  (b)  courses  and  instructional  programs  (20  articles),  and  (c) 
telecommunications  and  media  (13  articles).  Scriven  reported  no  significant  trends  in  these 
articles  other  than  work  on  course  attrition  (i.e.,  dropouts)  and  features  of  course  design.  More 
recently,  Koble  and  Bunker  (1997)  examined  the  abstracts  of  129  major  articles  from  The 
American  Journal  of  Distance  Education  published  between  1987  and  1995,  and  classified  them 
according  to  categories  offered  by  the  International  Centre  for  Distance  Learning.  The  results  of 
this  classification  are  presented  in  Table  5.  Within  the  media  and  delivery  systems  topic  of  the 
Koble  and  Bunker  (1997)  classification,  nine  articles  concerned  video  delivery,  nine  centered  on 
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computer  delivery,  five  involved  audioconferencing,  and  one  dealt  with  correspondence 
instruction.  The  others  were  various  combinations  of  media. 


Table  5. 

Classification  of  DL  Articles,  1987-95  (Koble  &  Bunker,  1997) 


Topic 

Number  of  Articles 

Percentage 

Theory,  policy,  and  development 

33 

26% 

Media  and  delivery  systems 
(Effectiveness/evaluation  methods) 

27 

21% 

Institution,  staff,  and  management 

20 

16% 

Student  psychology,  motivation 
and  characteristics 

19 

15% 

Faculty  participation  and  instructional 
process 

14 

11% 

Course  design  and  curriculum  development 

13 

10% 

Student  administration  and  support 

3 

.  2% 

A  review  of  the  characteristics  of  DL  in  different  settings  demonstrates  that  many  studies 
are  not  supported  by  an  adequate  experimental  design  and  do  not  offer  objective  measurement  of 
variables  (Walsh  et  al.,  1996).  Based  on  a  survey  of  129  organizations  involved  with  DL,  Walsh 
et  al.  (1996)  found  that  much  of  DL  is  simply  VTT  with  an  instructor  presenting  material  to 
students  at  remote  sites.  Figure  1  below  denotes  the  percentage  of  the  129  organizations 
reporting  the  use  of  a  particular  technology. 
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Percentage  of  Organizations  Using  Technology 
Figure  1.  Technologies  Used  in  Distance  Learning  (from  Walsh  et  ah,  1996) 
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Another  interesting  analysis  from  the  Walsh  study  was  the  type  of  research  conducted. 
Figure  2  charts  these  data  for  five  research  categories  from  the  125  organizations  that  responded 
to  this  question.  The  most  common  area  of  research  was  on  instructional  effectiveness,  which 
was  further  categorized  as  student  opinion  questionnaire  (59%),  instructor  opinion  questionnaire 
(39%),  comparative  test  results  (36%),  retention  data  (16%),  and  transfer  data  (16%). 


Effectiveness 
Use  of  various  technologies 
Multimedia  application 
Role  of  Instructor 
Curriculum  development 

0  20  40  60  80  100 

Percentage  of  Organizations  Examining  Topic 
Figure  2.  Areas  of  Distance  Learning  Research 

In  an  effort  to  develop  a  practical  user’s  reference,  Howard  (1997)  consolidated  available 
empirical  evidence  on  adult  education  and  DL  from  300  reports  published  between  1990  and 
1996.  Of  these  300  documents,  only  106  (35%)  were  considered  appropriate  based  on  source, 
subject  matter,  and  evaluation  methods.  These  articles  and  books  were  further  divided  into  ten 
categories  based  on  the  interests  of  the  study’s  sponsor  (the  Army).  Some  documents  accounted 
for  in  Table  6  were  included  in  more  than  one  category. 
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Table  6. 

Categories  ofDL  Studies,  1990-1996  (from  Howard,  1997) 


Topic _ _ _ Number 

Evaluation  of  DL  57 

VTT,  video  tapes,  interactive  videodisc  35 

Computer-based  training,  computer-mediated  conferencing  20 

Cost  effectiveness  and  systems  costs  18 

Student  interaction  with  instructors,  students,  and  technology  16 
Descriptions  of  specific  DL  programs  1 1 

Guidelines  for  planning  and  implementing  DL  9 

Professional  Education  7 

Reviews  of  DL  literature  7 

Miscellaneous  10 


FOCUS  OF  CURRENT  ANALYSIS 

Research  in  DL  has  been  exhaustively  documented  and  reviewed,  but  there  has  not  been 
a  systematic  examination  of  research  methodologies,  experimental  designs,  variables, 
comparison  groups,  sample  sizes,  and  other  research  factors  that  might  serve  as  the  scientific 
basis  for  establishing  DL  as  an  alternative  to  the  classroom.  This  report  examines  these  issues. 
Suen  and  Stevens  (1993)  reviewed  the  common  analytic  problems  and  errors  often  overlooked 
by  researchers  in  DL.  These  problems  and  errors  were  significance  testing,  assessment  issues 
(i.e.,  information  about  reliability  and  validity),  statistics  (e.g.,  error  rates,  practical  significance), 
and  reporting  practices  (e.g.,  demographics,  non-significant  findings,  degrees  of  freedom). 

For  our  analysis  of  research  on  training  using  a  DL  format,  the  broad  ranges  of  studies 
reported  in  the  literature  had  to  be  reduced  to  a  manageable  size.  We  first  performed  searches  of 
a  number  of  relevant  databases  (resulting  in  over  2,000  entries)  and  then  searched  the  World 
Wide  Web  for  relevant  facts  and  figures.  We  reviewed  the  1996,  1997,  and  1998  Proceedings  of 
the  Annual  Conference  on  Distance  Teaching  and  Learning  held  in  Madison,  Wisconsin,  for 
recent  contributions  from  this  significant  annual  meeting.  Our  analysis  of  the  papers  presented  at 
the  Madison  conference  indicated  that  8%  (16  out  of  200)  concerned  empirical  studies  of  DL, 
and  even  fewer  were  related  to  training.  Cross-references  from  other  review  articles  and  some 
key  books  on  distance  learning  were  also  examined  for  relevant  sources.  Our  goal  was  to 
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identify  a  representative  sample  of  evidence  that  was  restricted  to  training  but  focused  on  two 
key  ingredients  of  the  Moore  and  Kearsley  (1996)  definition  of  DL  —  instructional  techniques 
and  methods  of  communication. 

Since  our  focus  was  on  training,  we  restricted  subject  areas  from  academia  to 
methodological  and  technical  courses,  graduate-level  courses,  and  continuing  education 
programs  rather  than  introductory  or  lower-level  undergraduate  courses.  Studies  conducted  on 
K-12  populations  were  excluded  unless  they  clearly  contained  training  objectives,  such  as 
vocational  instruction  or  foreign  language  training  at  the  high  school  level.  Research  on  DL  in 
the  military  and  industry  was  acceptable  unless  it  was  oriented  to  educational  programs. 
Literature  reviews  and  planning  reports  discussing  research  and  training  issues  were  also 
included.  Our  selection  process  resulted  in  the  codification  of  43  research  reports,  seven  of 
which  were  reviews  of  research  literature  and  three  of  which  were  oriented  to  planning  for 
implementation  of  training  programs  that  included  research  and  evaluation  methods.  Of  the 
remaining  33  reports,  the  delivery  methods  are  described  below,  with  some  having  a 
combination  of  two  media: 

•  27  (82%)  included  VTT  as  a  communication  medium 

-  13  one-way  video/two-way  audio  -  2  videotape  exclusively 

-  6  two-way  video/two-way  audio  -  6  were  not  clearly  identified 

•  3  (9%)  were  audio  only  (including  audiographics) 

•  3  (9%)  included  computer-mediated  instruction 

•  6  (18%)  were  videotape  (in  combination  with  another  DL  medium) 

•  4  (12%)  included  print  (in  combination  with  another  DL  medium) 

•  1  (3%)  included  CBT  (in  combination  with  another  DL  medium) 

Of  the  33  reports,  31  reported  learning  outcome  data,  14  had  a  comparison  group 
(trainees  received  the  information  live,  rather  than  at  a  distance).  The  median  size  of  the  DL 
group  represented  in  a  single  report  was  n=106  (range  14  to  1,044  students)  and  the  median  size 
of  the  comparison  group  was  n=84  (range  18  to  401  students).  It  should  be  noted  that  some 
reports  included  several  iterations  of  the  same  course  or  had  several  classes  reported  as  a  single 
investigation.  When  this  was  taken  into  account,  the  average  “class  size”  for  a  separate  DL 
course  was  approximately  36  and  the  average  size  of  a  comparison  class  was  approximately  22. 
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Altogether,  the  database  represented  5,438  students  being  trained  through  DL  with  1,806 
comparison  students,  a  ratio  of  one  comparison  student  for  every  three  DL  students.  The  rank 
order  distribution  of  sample  sizes  for  the  DL  and  comparison  groups  are  charted  in  Figures  3  and 
4,  respectively. 


Rank  order 


Figure  3.  Sample  Sizes  of  the  DL  Groups  found  in  31  Studies  (rank  ordered  by  size) 
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Figure  4.  Sample  Sizes  of  the  Comparison  Groups  found  inl4  Studies  (rank  ordered  by  size) 
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Description  of  Empirical  Findings 

VTT  is  an  educational  application  derived  from  teleconferencing  which  allows  the 
delivery  of  training  occurring  in  a  classroom  to  remote  sites  (Wetzel,  Radtke,  &  Stem,  1994). 
The  essential  components  of  the  originating  classroom  are  the  instructor,  the  visual  images  being 
displayed,  and  audio.  Students  are  sometimes  present  at  the  origination  site;  a  two-way  video 
connection  which  allows  instructors  to  see  students  is  possible.  Typically,  there  is  an  audio 
return  from  the  remote  sites,  usually  through  a  telephone  connection.  Given  the  array  of  possible 
implementations,  the  general  conclusion  is  that  students  at  remote  sites  exhibit  either  no 
difference  or  only  a  slight  decrement  in  learning  when  compared  to  those  at  the  live  site  or  to  a 
comparison  group  in  a  traditional  classroom  (Chute,  Balthazar,  &  Poston,  1988;  U.S.  Office  of 
Technology  Assessment,  1989).  The  category  of  VTT  provided  the  overwhelming  number  of 
research  reports  for  our  analysis.  The  descriptions  of  the  reports  include  reported  information  on 
course  subject  and  length,  method  of  communication,  course  design  or  conversion  to  DL  from  a 
traditional  format,  the  DL  and  comparison  samples,  the  dependent  variables,  and  the  results. 

Military  Teletraining 

Bramble  and  Martin  (1995)  reported  on  the  Florida  Teletraining  Project,  which  tested  the 
feasibility  of  using  community  colleges  to  deliver  training  courses  to  military  reservists  with  the 
use  of  two-way  video/two-way  audio  technology.  The  project  developed,  delivered,  and 
evaluated  five  courses  for  personnel  at  five  remote  sites  in  three  states.  The  media  employed 
were  compressed  video  (256  Kbps)  transmitted  by  satellite,  a  document  camera  capable  of 
transmitting  images,  and  printed  student  study  guides. 

Each  of  the  five  courses  was  converted  from  previously  developed  courses.  The 
reconfiguration  to  a  DL  format  was  accomplished  within  a  six-month  period  (no  further  details 
were  provided).  A  common  methodology  in  the  course  design  was  the  use  of  word  pictures, 
which  provided  a  graphic  depiction  of  key  concepts  in  full  form  during  the  video  presentation, 
but  which  omitted  the  key  words  in  the  student  guides  (Cyrs  &  Smith,  1997).  The  courses, 
course  lengths,  and  sample  sizes  for  the  study  are  described  in  Table  7. 
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Table  7. 

Courses  and  Samples  in  the  Florida  Teletraining  Project  (Bramble  &  Martin,  1995) 
Course _ Course  Length _ Sample 


Unit  Administrative  Specialist 

73  hr. 

33 

Unit  Supply  Specialist 

96  hr. 

40 

Basic  Military  Police 

66  hr. 

26 

Handling  Hazardous  Waste 

6  hr. 

116 

Total  Oualitv  Leadershio 

6  hr. 

60 

Total  sample 

275 

Dependent  variables  were  standard,  criterion-based  proficiency  and  achievement  tests  for 
the  three  longer  courses  and  20-item  achievement  tests  for  the  two  short  courses.  Pre-  and 
post-tests  were  given  for  every  course  except  total  quality  leadership.  Overall  student  ratings  of 
the  course  components  were  collected  along  with  students  perceptions  of  the  course.  The  results 
are  displayed  in  Table  8.  The  technology  was  found  to  be  reliable  and  the  performance  tests 
indicated  solid  learning  gains.  No  comparison  group  was  included. 


Table  8. 

Comparisons  between  Live  and  DL  Instruction  (Bramble  &  Martin,  1995) 


Performance 

Reaction 

Course 

Measure 

%  pre-post  gain 

t-value 

“Effective  as  live 
instruction?”  %  ves 

Admin  Spec. 

Timed  Typing-wpm 

58.3% 

11.4* 

55% 

Type  Memo-errors 

-75.1% 

-7.0* 

- 

Supply  Spec. 

Achievement  test 

26.8% 

8.6* 

75% 

Milt.  Police 

Achievement  test 

28.9% 

13.6* 

76% 

Haz.  Waste 

Achievement  test 

64.1% 

21.5* 

84% 

TQL 

Achievement  test 

N/A 

81% 

*p  <.001,  one- 

-tailed  test 
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Research  in  the  Navy 

Wetzel  and  colleagues  (Simpson,  Wetzel,  &  Pugh,  1995;  Wetzel,  1996;  Wetzel,  Radtke, 
Parchman,  &  Seymour,  1996a;  Wetzel,  Pugh,  Van  Matre,  &  Parchman,  1996b)  evaluated  VTT 
courses,  with  two-way  audio,  delivered  by  the  Chief  of  Naval  Education  and  Training.  These 
courses  in  quality  assurance,  basic  leadership,  and  fiber  optic  cable  repair  were  delivered  to  Navy 
personnel.  In  each  of  these  studies,  no  differences  in  performance  were  found  between 
instruction  delivered  in  a  traditional  classroom  format  and  that  delivered  via  VTT  to  either  local 
or  remote  sites. 

Basic  leadership  training.  Simpson  et  al.  (1995)  described  an  existing  course  in  basic 
leadership  that  was  converted  to  a  VTT  format.  Four  instructors  taught  105  Navy  officers 
located  at  local  (n=36),  remote  (n=22)  and  traditional  classroom  (n=47)  sites  using  a  variety  of 
methods  including  lecture,  discussion,  experiential  learning,  and  team-building  exercises. 
Although  there  were  small  but  non-systematic  differences  between  the  DL  remote,  DL  local,  and 
traditional  classrooms  in  terms  of  student  perceptions,  participation,  and  the  observations  of 
subject  matter  experts,  there  were  no  significant  differences  between  locations  in  student 
knowledge  and  perceptions  of  training  and  learning  quality.  On  average,  students  performance 
on  a  25-item  multiple-choice  test  increased  22%  between  pre-  and  post-test  administrations.  All 
three  groups  —  comparison,  the  DL  local,  and  the  DL  remote  —  had  approximately  equivalent 
increases.  This  study  was  notable  in  its  use  of  multiple  measures  of  key  criteria.  For  example, 
student  participation  and  interactions  were  lower  in  the  two  DL  groups  based  on  an  objective 
count  of  questions  and  comments  by  observers,  subject  matter  expert’s  observations,  and  self- 
report  by  students. 

Fiber  optic  cable  repair  training.  Wetzel  et  al.  (1996),  examined  50  students  who  were 
instructed  over  five  days  with  a  structured  format  of  lecture,  computer-based  training, 
demonstrations,  laboratories,  homework  reviews,  and  question  and  answer  periods.  Students 
were  approximately,  but  not  randomly,  divided  among  two  DL  groups  (DL  local  and  DL 
remote),  and  a  comparison  (non-DL)  group.  The  scores  on  the  course  final  exam  were  slightly 
higher  in'the  comparison  group  (86%  correct)  than  in  the  DL  local  (85%)  and  DL  remote  (80%) 
groups,  but  this  difference  was  not  statistically  significant.  Although  it  took  students  longer  at 
the  remote  site  to  complete  their  lab  assignments,  there  were  no  significant  differences  between 
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the  groups  in  terms  of  procedural  errors,  observer  ratings  of  safety,  quality  of  work,  or  objective 
errors. 

Quality  assurance  training.  Similarly,  Wetzel  et  al.  (1996b)  found  no  mean  differences 
in  final  examination  scores  among  233  students  taking  a  course  in  quality  assurance  at  a  local 
(94%  correct),  remote  (92%),  and  comparison  site  (94%).  A  second  measure  of  performance 
was  administered  to  students  in  the  local  and  remote  DL  groups.  Average  scores  on  this  end-of- 
course  10-item  quiz  on  computer  procedures  were  nearly  identical  (46%  vs.  47%  correct). 

Celestial  navigation.  Wetzel  (1996)  performed  the  evaluation  of  a  refresher  course  in 
celestial  navigation.  Students  (n=279)  across  two  DL  groups  (remote  and  local)  and  a 
comparison  group  were  compared  on  performance,  reaction  measures,  and  amount  of  interaction 
as  determined  by  an  observer.  There  were  no  significant  differences  among  the  DL  groups  on 
students’  homework  scores,  but  students  in  the  remote  group  scored  slightly,  but  statistically, 
lower  on  their  final  examinations  than  students  in  the  local  group.  When  inequities  in  seniority 
status  were  controlled  in  this  data,  students  in  the  remote  condition  still  scored  4%  lower  than 
those  in  the  local  site.  The  use  of  picture-in-picture  technology  received  overwhelmingly 
positive  responses  from  students  at  the  remote  site,  but  students  at  the  local  site  were  ambivalent 
about  this  technology. 

One-way  Video  /  Two-way  Audio  Media 

Although  the  courses  using  VTT  to  this  point  involved  two-way  video  and  two-way 
audio,  Crawford  and  Suchan  (1996)  argued  that  one-way  video  and  two-way  audio  systems  are 
far  less  costly  and  just  as  rich  a  communication  medium  when  learning  outcomes  require  only 
verbal  interaction  between  students  and  instructors.  They  make  a  convincing  argument  that  the 
choice  of  media  used  should  be  determined  based  on  the  necessary  learning  outcomes  and 
instructional  technique.  Of  course,  the  availability  of  the  necessary  media  might  be  a  deciding 
factor. 

New  product  training.  Souder  (1993)  delivered  the  same  semester-long  course  on 
managing  new  product  innovations  to  24  students  via  one-way  video/two-way  audio  technology, 
13  students  live  in  the  broadcast  studio  (who  were  able  to  interact  with  the  DL  group)  and  20 
students  at  a  separate  time  in  a  traditional  classroom.  No  information  was  provided  on  course 
design  or  conversion.  All  students  took  the  same  essay  examination  and  completed  a  research 
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term  paper.  Students  in  the  DL  group  scored  significantly  higher,  on  average  (95%  correct),  than 
students  in  the  comparison  broadcast  studio  group  (92%)  but  not  higher  than  those  in  the 
comparison  traditional  group  (94%).  Students  in  the  DL  group  also  received  significantly  higher 
grades  on  their  case  study  analyses  (95%)  than  students  in  the  broadcast  studio  group  (85%). 
There  was  no  difference  in  performance  on  the  research  term  papers  among  the  three  groups.  As 
the  author  notes,  however,  the  three  groups  of  students  differed  dramatically  in  terms  of  age, 
grade  point  average,  and  industrial  experience,  with  those  in  the  DL  group  significantly  higher 
on  all  three  factors. 

Air  traffic  quality  assurance  training.  Lennon  and  Pa)me  (1997)  examined  the 
differences  in  student  reaction  and  learning  outcomes  between  DL  (one-way  video/two-way 
audio)  and  comparison  non-DL  groups  of  Air  Traffic  Quality  Assurance  Specialists  of  the 
Federal  Aviation  Administration.  The  7  ‘/a-day  course  (“Quality  Assurance  Program 
Administration”)  was  delivered  on  two  occasions  by  four  instructors  to  a  total  of  31  employees 
in  the  non-DL  group  and  delivered  by  a  trained  instructor  to  18  employees  at  four  sites  (DL 
group).  There  were  no  significant  differences  on  any  of  the  12  reaction  measures,  which  were 
rated  on  a  5-point  scale  and  included  pace  of  training,  relevance  to  job,  effectiveness  of 
instructors,  and  overall  quality  between  the  two  groups.  In  addition,  there  were  no  differences 
between  the  groups  on  either  the  pre-test  or  post-test  knowledge  measures  that  consisted  of  50- 
item  multiple  choice  open  book  tests. 

Health  care  providers.  An  overlooked  population  of  individuals  in  distance  education  is 
the  elderly.  Coogle,  Osgood,  Parham,  Wood,  and  Churcher  (1996)  addressed  this  gap  by 
evaluating  a  teleconference  on  the  topic  of  geriatric  alcoholism  which  targeted,  among  others, 
older  adults  and  their  family  caregivers.  This  two-hour  program  was  delivered  to  134 
participants  via  one-way  video/two-way  audio  technology.  It  consisted  of  a  panel  of  four  experts 
who  presented  information  on  topics  such  as  signs  and  symptoms  of  alcoholism  and  differences 
between  younger  and  older  alcoholics.  In  addition,  the  panel  used  live  dramatization,  case 
studies,  and  answered  questions  phoned  in  by  participants. 

The  same  23  questions  were  given  both  as  a  pre-test  and  a  post-test  on  the  course  concepts. 
As  a  whole,  the  participants’  knowledge  significantly  increased  between  the  two  administrations 
(pre-test:  67%  correct;  post-test:  82%  correct).  This  pattern  of  increase  was  similar  for  both  the 
elderly  residents  (n=9)  and  the  service  providers  (n=87).  The  researchers  concluded  that  the 
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interactive  teleconference  was  an  effective  vehicle  for  delivering  education  to  older  adults. 
However,  without  the  benefit  of  a  comparison  group,  it  is  difficult  to  determine  if  the  gains  in 
knowledge  were  due  to  other  factors,  such  as  simple  memory  effects  or  whether  the  same  gains 
could  have  been  achieved  by  having  participants  read  a  pamphlet  with  the  necessary  information 
rather  than  delivering  the  information  via  distance  education  technology. 

Air  trajfic  control  and  knowledge  retention.  In  a  study  that  focused  on  the  effects  of 
knowledge  decay  over  extended  periods  during  a  distance  learning  course.  Wisher,  Seidel,  Priest, 
Knott,  and  Cumow  (1997)  examined  air-traffic-control  operators  distributed  to  Army  National 
Guard  soldiers  (n=32)  at  eight  remote  sites  over  an  11  month  period.  The  media  employed  was 
one-way  video/two-way  audio  supplemented  with  study  guides  and  the  training  manuals  used  in 
the  resident  course.  The  course  had  six  phases:  the  first  four,  which  were  taught  through  DL, 
were  knowledge-oriented  on  topics  such  as  aviation  weather,  or  rules  and  principles  for  radar  and 
tower  operations;  the  final  two  phases  were  taught  in  residence  at  Fort  Rucker,  Alabama,  and 
were  hands-on,  skill-based  training  on  radar  and  tower  operations. 

The  course  was  converted  from  the  resident  version,  primarily  by  the  instructor,  for  the 
DL  course  (no  details  on  time  or  costs  were  provided).  The  soldiers  in  the  comparison  group 
(n=45)  attended  all  six  phases  of  the  course  for  1 1  weeks  at  Fort  Rucker.  The  dependent 
variables  were  a  controlled  exam  on  fundamental  control  tower  procedures  administered  by  the 
Federal  Aviation  Agency,  a  knowledge  retention  test,  achievement  test  scores,  performance 
scores,  and  completion  rates.  The  knowledge  retention  test  concerned  topics  trained  in  the  first 
phase.  Due  to  course  schedules,  it  was  administered  20  weeks  later  for  the  DL  group  and  10 
weeks  later  for  the  comparison  group.  The  achievement  test  scores  measured  the  four 
knowledge  phases;  performance  scores  measured  the  two  skill-based  phases  of  training. 

The  results  indicated  that  the  DL  group  performed  as  well  as,  but  not  better  than,  the 
comparison  group  on  the  knowledge  phases  of  the  training.  For  knowledge  retention,  the  DL 
group  dropped  14%  over  20  weeks  in  contrast  to  a  15%  drop  in  the  comparison  group  after  10 
weeks.  This  drop  was  similar  to  that  found  by  Semb  and  Ellis  (1994)  who  determined  that  in  23 
studies  of  knowledge  retention  over  periods  of  20  weeks  or  less,  the  relative  loss  of  knowledge 
was  13.7%.  On  the  basis  of  Wisher  et  al.  (1997),  training  through  distance  learning  was  retained 
as  well  as  conventional  classroom  instruction.  The  major  difference  was  in  performance  in  the 
two  skill  phases  where  the  DL  group  had  a  success  rate  of  58%  on  the  tower  laboratory 
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compared  to  a  success  rate  of  90%  for  the  comparison  (non-DL)  group.  For  the  radar  laboratory, 
the  DL  group  had  only  a  14%  success  rate  compared  to  an  85%  success  rate  for  the  comparison 
group.  The  authors  believed  that  while  the  facts  and  declarative  knowledge  were  well  retained, 
the  retention  interval  of  three  months  between  learning  and  applying  the  rules  and  principles  was 
too  lengthy.  This  did  not  allow  an  immediate  transition  to  applying  these  rules  and  principles  in 
the  laboratory,  where  speed  and  accuracy  of  responses  were  critical.  The  comparison  group  was 
able  to  immediately  apply  that  knowledge,  as  their  phases  were  contiguous. 

Engineering  training  off-campus.  In  a  study  on  engineering  training  at  the  graduate  level. 
Stone  (1988)  examined  the  characteristics  of  students  enrolled  in  off-campus  degree  and 
continuing  education  programs  to  on-campus  students.  The  semester  long  courses  were  in  the 
engineering  field.  Data  were  collected  over  a  five-year  period.  The  DL  group  received  either  a 
one-way  video/two-way  audio  satellite  transmission  or  a  videotaped  version  of  the  lectures;  the 
comparison  group  received  the  same  lectures  in  the  classroom  (classrooms  were  equipped  with 
satellite  transmission  and  video  recording  capabilities). 

The  DL  sample  contained  off-campus  students  (n=726)  and  the  comparison  group  was 
composed  of  students  (n=302)  attending  the  lectures  face-to-face  with  the  instructors.  The 
dependent  variable  was  grade  point  average  (GPA).  The  results  of  an  analysis  of  variance 
indicated  that  the  DL  group  for  the  degree  program  had  achieved  a  significantly  higher  GPA  than 
the  on-campus  comparison  group  (3.44  versus  3.25);  the  continuing  education  students  had  a 
GPA  of  3.21.  This  did  not  have  a  direct  comparison  to  an  on-campus  program  but,  in  view  of  the 
sample,  would  likely  equate  to  the  on-campus  degree  group.  These  data  clearly  support  the 
supposition  that  VTT  and  videotape  (in  combination  with  the  printed  text)  were  equally  effective 
in  training  technical  knowledge  and  skills. 

VTT  evaluation  techniques.  In  an  analysis  of  one-way  video/two-way  audio  delivered  by 
satellite.  Wisher  and  Cumow  (1998)  sought  to  develop  a  single-page  evaluation  instrument  for 
short-term  training  events,  defined  as  those  occurring  within  a  single  day.  Eight  separate  training 
events  were  analyzed.  Six  were  oriented  to  military  training  topics,  such  as  risk  management 
and  airborne  call  for  fire,  and  two  were  oriented  to  more  general  civilian  topics. 

The  courses  did  not  have  built-in  performance  evaluation  measures  such  as  an  end-of- 
course  exam  or  a  proficiency  test.  As  a  surrogate  learning  outcome  measure,  the  researchers 
applied  a  two-part  self  assessment  scale  that  asked  first  whether  or  not  the  student  had 
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participated  in  a  previous  course  on  the  topic.  It  then  asked  “compared  to  what  you  already 
knew  about  topics  related  to  (today’s  training  topic),  how  much  more  did  you  learn  from  this 
training  event”  (Wisher  &  Cumow,  1998,  p.  19).  Previous  research  has  demonstrated  the  utility 
of  such  assessments  for  military  personnel,  where  training  is  conducted  continually  and 
performance  is  frequently  assessed  on  a  dichotomous  GO  or  NO  GO  basis  with  prompt 
feedback.  The  courses  were  either  originally  developed  for  the  broadcasts  or  were  largely  intact 
from  existing  courses  with  minimal  conversion  required.  The  DL  sample  was  comprised  of 
members  of  the  Army  National  Guard  (n=l,044).  No  comparison  group  was  included.  The 
dependent  variables  were  the  self-assessed  prior  knowledge  and  learning  described  above.  There 
were  also  a  series  of  nine,  questions  related  to  the  quality  of  the  technology  and  perceptions  of  the 
instructor  (e.g.,  quality  of  video,  responsiveness  to  questions). 

The  results  of  the  surrogate  learning  outcome  measure  are  presented  in  Table  9  for  the 
first  iteration  of  a  one-day  risk  management  course.  Note  that  the  simple  variable  of  having  had 
or  not  had  a  previous  course  on  the  training  topic  clearly  distinguished  the  two  groups,  with 
nearly  twice  as  many  who  had  no  previous  course  reporting  that  they  learned  a  lot  from  the 
training.  This  pattern  of  greater  learning  held  in  each  of  the  six  military  training  events  and  was 
significantly  different  as  tested  by  an  analysis  of  variance  (p  <  .001).  This  supports  the  face 
validity  of  the  measure.  Within  the  three  iterations  of  the  risk  management  course,  an  ANOVA 
revealed  no  significant  differences  among  the  three  events,  which  lends  support  to  the  reliability 
of  the  measurement  technique. 

Table  9. 

Analysis  of  Amount  Learned  versus  Previous  Course  Experience  in  the  First  Iteration  of  the 
Risk  Management  Course  (n=202)  (Wisher  &  Cumow,  1998) 

No  Previous  Course  Previous  Course 


Learned  “a  little”  (1  or  2) 

6% 

22% 

Learned  “some”  (3) 

32% 

46% 

Learned  “a  lot”  (4  or  5) 

64% 

32% 
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The  results  of  the  technology  and  instruction  ratings  are  displayed  in  Table  10.  As  a 
comparative  benchmark,  the  technology  ratings  described  in  Wetzel,  et  al.  (1996)  which  has 
substantially  similar  questions  and  used  the  same  scale  are  displayed. 


Table  10. 

Comparison  Mean  Ratings  for  Navy  Benchmark  (Wetzel,  et  al.,  1996)  and  ARNG  study 
(Wisher  &  Cumow,  1998) 


Question 

Navy 

Benchmark 

Army  Events 

M  SD  n 

Location  of  video  screen 

4.6 

4.4 

.81 

1031 

Quality  of  audio 

4.1 

3.8 

1.21 

1023 

Quality  of  video 

4.5 

4.2 

.96 

1025 

Instructor  effectiveness 

— 

3.9 

.91 

960 

Opportunity  to  ask  questions 

— 

3.7 

1.22 

903 

Responsiveness  to  student  questions 

4.5 

3.9 

1.00 

849 

Relevance  of  course  to  guard  duties 

4.3 

4.1 

.96 

938 

Overall  learning  environment 

— 

4.0 

.99 

1005 

Overall  effectiveness  of  instruction 

— 

3.8 

.98 

1000 

Military  command  training.  In  a  study  by  Keene  and  Cary  (1990),  the  second  phase  of  a 
course  entitled  Command  and  General  Staff  Officers’  Course  was  offered  through  one-way 
video/two-way  audio  broadcast.  A  total  of  145  Army  reserve  officers  (n=  145)  participated  in  the 
69-hour  course  over  a  two  week  period.  The  DL  instruction  consisted  of  41  of  the  69  class  hours 
(59%),  where  21  hours  were  dedicated  to  lecture  and  discussion  and  20  were  used  for  practical 
exercises.  The  course  design  was  modified  from  the  traditional  counterpart  differing  only  in  the 
video  broadcast  method  of  instruction.  No  additional  details  were  provided. 

This  study  included  two  DL  groups  (n  =  36  and  n  =  26)  and  one  comparison  (non-DL) 
group  (n  =  38).  The  dependent  variables  were  a  pre-test  and  post-test  of  basic  skills.  The  pre-test 
consisted  of  multiple  choice  items  from  a  comprehensive  skills  exam,  and  the  post-test  included 
measures  of  recognition  and  recall  learning.  All  students  were  taking  the  class  for  professional 
development  and  were  similar  on  all  demographic  variables. 
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The  measures  of  learning  showed  a  difference  in  pre-test  scores  between  the  DL  and 
comparison  groups.  A  multivariate  analysis  of  variance  was  used  to  compare  pre-test  to  post-test 
scores.  Taking  the  original  differences  into  consideration,  the  DL  group  received  significantly 
higher  scores  for  three  of  the  four  measures. 

Telecourse  evaluation  questionnaire,  Bink,  Biner,  Huffman,  Geer  and  Dean  (1995) 
conducted  a  study  on  students  taking  college-level  continuing  education  courses  delivered 
through  one-way  video  and  two-way  audio.  Eighteen  different  courses  of  unspecified  content 
were  offered  at  68  off-site  locations  throughout  Indiana.  A  total  of  n  =  106  students  were 
included  in  the  DL  group.  There  was  no  information  on  course  development  or  whether  they 

were  converted  from  previously  existing  courses. 

The  dependent  variables  include  attitudinal  and  course-related  items  as  well  as  a  traditional 
predictor  item  (prior  GPA).  The  Telecourse  Evaluation  Questionnaire  was  used  to  assess  students’ 
satisfaction  on  a  5-point  Likert  scale.  Several  independent  variables,  such  as  students  year  in 
college,  prior  experience  with  telecourses  and  current  course  load  ratings  were  considered. 
Additional  demographic  information  was  considered  as  well. 

Final  course  grade  was  significantly  correlated  with  prior  GPA,  ratings  of  satisfaction,  and 
number  of  years  in  college.  Using  a  multiple  regression  analysis,  the  authors  found  that  prior  GPA 
was  a  significant  predicting  variable  for  final  course  grade.  Student  ratings  of  promptness  and 
number  of  years  in  college  accounted  for  variance  in  final  course  grade  that  was  unexplained  by 
prior  GPA.  Demographic  variables  were  not  associated  with  final  course  grade  in  either  analysis. 
Interestingly,  ratings  of  satisfaction  were  significantly  correlated  with  learning  outcome,  but  much 
less  than  prior  GPA. 

Mathematics  training,  Larson  and  Pruning  (1996)  reported  a  study  on  a  pre-college 
mathematics  course  that  was  equivalent  to  a  pre-calculus  course  but  was  designed  for  those 
students  who  would  not  be  taking  calculus.  The  course  was  offered  for  a  total  of  33  weeks  and 
consisted  of  classes  30  minutes  in  length  with  meetings  three  times  a  week.  Through  the  use  of 
one-way  video/two-way  audio,  91  lessons  were  offered  via  satellite.  The  remaining  66  lessons 
were  supported  by  lesson  guides  (which  included  but  were  not  limited  to  activities,  assignments, 
teaching  aids,  etc.).  Twenty-one  DL  sites  were  included  in  the  analysis  with  a  total  sample  of 
n=102  students.  The  comparison  sample  consisted  of  102  students  enrolled  in  a  traditional  pre¬ 
calculus  course.  The  DL  course  was  created  specifically  for  the  purpose  of  teaching  students 
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who  were  considering  furthering  their  education  beyond  high  school  but  did  not  plan  to  take 
calculus  courses.  This  DL  course  used  textbooks,  curriculum  guides,  and  satellite  broadcast 
videos  in  accordance  with  the  multimedia  recommendations  by  Pelton  (1991). 

An  analysis  of  qualitative  student  perceptions  of  the  DL  format,  curriculum,  instmction, 
and  students’  disposition  found  that  although  most  students  reported  enjoying  the  DL  course,  the 
majority  of  them  would  have  preferred  the  traditional  format.  In  addition,  the  comparison  group 
performed  significantly  better  than  the  DL  group  on  both  the  pre-test  and  post-test  of  the  placement 
test.  This  is  not  surprising  since  the  classes  focused  on  different  types  of  students.  Also  of  interest 
is  that  the  percentage  of  students  in  the  DL  group  at  or  above  the  appropriate  level  increased  from 
the  beginning  of  the  DL  course  (from  51%  to  65%)  whereas  the  percentage  of  students  in  the 
traditional  course  at  or  above  the  appropriate  level  decreased  (from  90%  to  77%).  The  satellite 
course  resulted  in  more  correct  placement  of  students  in  a  mathematics  course  than  did  the  non-DL 
course. 

Information  systems  training.  In  a  study  that  explored  the  application  of  a  distributed 
group  support  system  in  the  delivery  of  distance  education.  Fellers  and  Moon  (1994)  examined 
an  upper-level  undergraduate  course  on  information  systems  that  was  15  weeks  in  length.  The 
methods  of  communication  were  two-way  video/two-way  audio  over  a  fiber  optic  cable  network. 
The  instruction  also  used  videotapes  and  a  document  camera  for  displaying  graphics  and  three- 
dimensional  objects.  No  additional  information  was  provided  regarding  conversion  to  a  distance 
learning  format. 

The  DL  group  was  n=24  undergraduates.  No  comparison  group  was  provided.  The 
dependent  measure  was  a  midterm  and  end  of  course  survey  on  student  perceptions.  The 
researchers  sought  to  determine  whether  student  opinions  towards  distance  learning  change, 
either  improving  as  they  become  accustomed  to  the  technology  or  decrease  for  any  number  of 
reasons.  The  findings  from  a  matched  t-test  showed  only  one  of  nine  items  to  be  significantly 
different,  namely  a  decrease  in  the  visuals  being  clear  and  easy  to  read  but  probably  due  to  the 
one  technical  failure  (faulty  monitor)  that  occurred  on  the  day  of  the  second  survey.  There  has 
been  concern  by  some  in  the  DL  field  that  as  students  become  more  accustomed  to  the  delivery 
through  electronic  media,  they  will  be  more  demanding  of  its  quality.  On  the  basis  of  this 
limited  study,  it  appears  not  to  be  the  case  within  the  restrictions  of  a  15-week  course. 
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Learner  satisfaction.  Biner,  Dean,  and  Mellinger  (1994)  conducted  two  investigations  to 
identify  the  dimensions  of  learning  satisfaction  with  a  one-way  video/two-way  audio  system. 
The  first  investigation  concerned  14  televised  courses  to  43  off-campus  locations  and  a  mixture 
of  graduate  and  undergraduate  students  (n=201);  the  second  investigation  concerned  13  courses 
to  36  off-campus  locations,  also  with  a  mixture  of  students  (n=177).  No  information  was 
supplied  on  the  course  content,  design,  or  whether  it  had  been  converted  from  an  existing  course. 

The  purpose  of  the  study  was  to  explore  the  factor  structure  of  an  evaluation  instrument 
developed  by  Biner  (1993).  The  second  investigation  served  as  a  confirmatory  analysis.  A 
factor  analysis  was  performed  on  responses  to  33  items,  resulting  in  the  extraction  of  seven 
factors  that  accounted  for  71%  of  the  data’s  variance  in  both  investigations.  The  seven 
dimensions  identified  are  listed  in  Table  11  below. 

Table  11. 

Seven  Dimensions  Underlying  DL  Student  Satisfaction  (Biner  et  al.,  1994) 


Dimension 

Item  Loading  on  Factor 

Instructor/instruction 

.93 

At-site  personnel 

.88 

Technology 

.85 

Support  services 

.76 

Promptness  of  material  delivery 

.73 

Course  management 

.65 

Communication  with  instructor  (out  of  class) 

.63 

Social  presence  factors.  In  research  on  the  effect  of  perceived  social  presence  on  student 
outcomes  in  VTT  and  face-to-face  classes,  Gunawardena  and  Zittle  (1997)  examined  whether 
social  presence  is  a  predictor  of  satisfaction.  The  study  involved  an  academic  exercise  that 
occurred  over  the  course  of  a  semester.  The  communication  medium  was  computer-mediated 
conferencing. 

The  academic  exercise  provided  a  forum  for  graduate  students  at  five  universities  to  share 
and  discuss  experiences  with  DL  research.  Participation  in  the  exercise  was  a  course 
requirement.  There  was  no  mention  whether  the  course  had  been  previously  taught  in  a 
traditional  classroom  and  later  converted  to  VTT.  (N.B.  Given  the  nature  of  the  course  and  a 
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requirement  for  collaborative  learning,  the  course  was  probably  designed  solely  for  VTT).  The 
DL  group  was  n=50  students.  No  comparison  group  was  presented. 

The  dependent  measures  were  embedded  in  a  61-item  questionnaire  administered  upon 
course  completion.  Examples  of  variables  of  interest  were  social  presence,  active  participation 
in  the  conference,  technical  training  received  at  site,  and  capability  of  mastering  computer- 
mediated  communication,  the  last  two  being  the  closest  to  a  self-assessment  of  learning.  The 
overall  results  indicated  that  social  presence  is  a  strong  predictor  of  satisfaction  in  computer- 
mediated  communication.  The  two  learning-related  variables  described  above  scored  3.2  and  4.2 
respectively  on  a  5-point  scale  which  appears  to  reflect  moderate  satisfaction  with  the  technical 
training  received  at  the  site  and  reasonable  satisfaction  with  the  capability  to  master  the 
computer-mediated  communication  technology. 

Computer  Conferencing 

Computer  conferencing  offers  students  the  opportunities  to  interact  with  each  other  and 
their  instructor  over  computer  networks  (Moore  &  Kearsley,  1996).  These  interactions  occur 
through  electronic  mail,  bulletin  boards,  chat  rooms,  file  transfers,  or  the  transfer  of  images  that 
can  be  annotated  by  means  of  whiteboards.  These  interactions  can  occur  in  real  time 
(synchronous)  or  on  a  delayed  basis  (asynchronous).  There  have  been  several  studies  that 
reported  the  comparative  effectiveness  of  computer  conferencing  in  a  training  context,  and  many 
others  that  offered  descriptive  case  studies  and  prescriptive  overviews  consisting  of  message 
traffic  counts  between  student  and  instructor. 

Military  engineer  traihing.  In  a  study  reported  by  Phelps,  Ashworth,  and  Hahn  (1991), 
computer  conferencing  was  included  in  the  delivery  of  training  materials  to  reserve  officers  as 
part  of  a  course  for  the  Engineer  Officer  Advance  Course.  The  DL  component  was  one  two- 
week  module  from  a  seven  module  Engineer  Officer  Advance  Course.  The  materials  used  in  the 
resident  course  were  converted  to  the  DL  format  with  the  breakout  described  in  Table  12. 
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Table  12. 


Percentage  of  Resident  Course  Materials  Converted  to  DL  Format  (Phelps  et  ah,  1991) 


Media  Task 

Percentage 

Print 

41% 

Team  Asynchronous 

20% 

Computer-based  Instruction 

19% 

Quiz/Exam/Review 

14% 

Team  Synchronous 

4% 

Video 

2% 

100% 


The  two-week  module  covered  topics  such  as  flexible  pavement  structures,  asphalt 
production,  and  petroleum  pipelines.  The  conversion  of  the  66-hour  module  to  the  DL  format 
required  4,250  labor  hours  as  described  in  Table  13. 


Table  13. 

Estimated  Time  for  Course  Conversion  (Phelps  et  al.,  1991) 


Staff  Hours 


%  Effort 


.7 - - - 

Course  Requirements  Analysis 

435 

10% 

Course  Design 

163 

4% 

Course  Development 

2589 

61% 

CBI  /  Slide  Conversion 

812 

19% 

Video  Taoe  Production 

251 

6% 

Fourteen  reservists  served  as  the  DL  group  and  the  comparison  group  was  constituted 
from  final  exam  scores  (n=339)  at  the  resident  site  as  well  as  a  subset  of  resident  students  (n=49) 
for  purposes  of  assessing  demographics  and  perceptions  at  the  resident  site.  Dependent  variables 
for  the  course  were  pre-  and  post-course  student  perceptions  of  their  knowledge  of  the  course 
topics,  test  scores,  and  course  completion  rates. 

There  were  no  demographic  differences  between  the  DL  group  and  the  comparison  subset 
resident  group.  For  learning  outcomes,  differences  between  the  test  scores  of  the  DL  students 
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were  not  significantly  different  from  the  students  in  residence.  However,  the  comparison  of 
students’  self-assessment  of  their  knowledge  before  and  after  the  course  showed  that  the  DL 
group  had  a  significant  gain  over  the  resident  group  (33%  versus  12%).  There  was  a  higher 
course  completion  rate  for  the  residence  group  (95%  versus  64%  for  DL)  with  conflicts  due  to 
family  and  job  cited  as  the  reasons  for  attrition  from  the  course. 

Audio  Technology 

The  importance  of  transmitting  high  quality  audio  for  facilitating  DL  instruction  has  been 
identified  numerous  times  in  the  literature  (Garrison,  1990;  Hardy  &  Olcott,  1995).  This  is  true 
for  audio/print  only  instruction  as  well  as  audio  coupled  with  one-way  or  two-way  video  signals 
in  a  VTT  setting.  The  audio  technology  category  includes  audiographics  systems,  which  employ 
electronic  whiteboards  for  the  instructor  to  mark  in  useful  ways  and  also  as  application  sharing 
software  which  allows  computer-displayed  images  to  be  transmitted  to  remote  sites  over 
telephone  lines  (as  well  as  across  the  Internet).  Audio  represents  the  most  common  and  least 
expensive  form  of  synchronous  interactive  teletraining. 

Unit  clerk  training.  In  a  study  of  the  cost  effectiveness  of  audio  teletraining  for  the  Army 
National  Guard,  Wisher  and  Priest  (1998)  report  on  a  three  week  unit  clerk  course  delivered 
through  study  guides  and  audio  conferencing.  Students  in  the  DL  group  assembled  at  armories 
in  the  geographic  vicinity  to  participate  in  a  five  and  a  half  hour  daily  training  session  using  an 
audio  bridge  which  connected  between  eight  and  ten  remote  sites  (each  with  about  six  students) 
to  the  central  training  facility. 

The  course  provided  instruction  on  47  clerical  tasks  such  as  preparing  personnel 
qualification  records  or  determining  qualification  for  promotion.  Of  these  tasks,  16  were 
declared  critical  and  performance  tests  were  administered  throughout  the  three-week  program. 
There  were  no  course  conversion  issues  as  the  same  instructors  delivered  the  same  course  to  the 
DL  and  comparison  groups.  The  only  difference  in  training  was  the  audio  teletraining  students 
could  hear,  but  not  see,  their  instructors. 

Three  iterations  of  the  DL  course  resulted  in  a  total  sample  of  1 18  students;  the 
comparison  group  consisted  of  a  sample  of  107  students.  Dependent  variables  were  performance 
on  a  diagnostic  test,  performance  on  the  first  test  after  training  on  the  16  critical  tasks,  and 
completion  rates.  The  data  are  presented  in  Table  14. 
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Table  14. 


Results  of  Audio  Teletraining  (from  Wisher  &  Priest,  1998) 


Dependent  Measure 

DL  Group 
Pass  Rate 

Comparison 
Pass  Rate 

t-value 

Diagnostic  test 

9% 

11% 

not  significant 

First  test  after  training 

86% 

94% 

4.74* 

Graduation  rate 

100% 

100% 

—  ' 

*  p  <.001,  one-tailed  test 

The  DL  group  had  a  slight  advantage  in  learning  performance  the  first  time  around  (94% 
versus  86%  pass  rate),  but  the  overall  graduation  rate  was  100%  for  both  groups.  Since  the  DL 
group  avoided  the  costs  to  travel  and  be  billeted  at  the  central  training  site,  there  was  a  cost 
avoidance  of  over  $1,000  per  student  in  the  audio  teletraining  group.  This  translates  into  savings 
of  nearly  $300,000  annually  if  the  course  were  to  be  delivered  entirely  by  audio  teletraining. 

Summary  of  Reviewed  Studies 

The  twenty  studies  reviewed  portray  a  generally  positive  view  of  the  effectiveness  of  DL 
for  training  applications.  Our  selection  process  resulted  in  the  review  of  studies  whose  reported 
methods  of  communication  parallel  that  of  academia,  industry  and  government  —  for  example, 
80%  of  our  sample  reported  VTT  use  in  comparison  to  81%  for  the  Walsh  et  al.  (1996)  review  of 
the  characteristics  of  DL  providers.  We  will  now  assess  our  findings  at  a  global  level  and 
comment  on  how  well  they  address  our  goals  in  reviewing  the  research  literature. 

Our  interest  was  in  appraising  the  effects  of  three  identifying  features  of  DL  on  the 
learning  outcomes.  The  three  principal  features,  as  offered  in  Moore  and  Kearsley’s  (1996) 
definition  of  DL,  were  course  design,  instructional  techniques,  and  methods  of  communication. 
On  the  basis  of  this  review,  we  offer  the  following  assessment  of  the  quality  of  the  research 
literature.  We  categorized  the  degree  to  which  each  of  the  20  training  reports  (a)  Fully 
described,  (b)  Partially  described,  or  (c)  Did  not  mention  (or  relying  on  inferences)  the  three  key 
features  of  DL.  The  results  of  our  assessment  are  presented  in  Table  15. 
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Table  15. 

Assessment  of  Completeness  of  Information  in  Research  Reports 


Course  Design  / 
Conversion 

Instructional 

Techniques 

Methods  of 
Communication 

Fully  described 

30% 

30% 

50% 

Partially  described 

30% 

45% 

50% 

Not  mentioned  -  inferred 

40% 

25% 

Clearly  there  are  problems  with  the  completeness  of  the  documentation.  The  descriptions 
of  ‘methods  of  communication,’  which  describe  the  technology  that  link  two  or  more  sites,  was 
in  the  best  condition,  as  all  reports  mentioned  the  technology  which  was  used.  The  ‘instructional 
technique’  descriptions  left  something  to  be  desired  with  only  about  a  third  giving  a  full 
description  but  one-fourth  not  mentioning  how  instruction  occurred.  The  course  design  or  course 
conversion  element  was  in  the  poorest  condition,  with  40%  not  mentioning  what  was  done  to 
create  or  modify  a  course.  In  some  cases,  we  were  able  to  make  inferences  that  a  course  was 
apparently  newly  designed  or  was  clearly  converted  from  an  existing  course.  Not  knowing  more 
specific  details  on  these  key  issues  makes  judgments  difficult  -  was  it  the  instructional 
technique,  media,  or  great  effort  in  converting  and  improving  an  existing  course  to  a  DL  format 
that  accounts  for  the  improvement  beyond  a  comparison  group?  Clark  (1989)  has  often  made 
this  point  in  his  analysis  of  comparing  multimedia-based  instruction  to  the  traditional  classroom. 

CONCERNS  IN  EXPERIMENTAL  DESIGN 
The  studies  reviewed  in  this  report  can  be  classified  in  terms  of  the  research  design  used  by 
the  experimenters  to  determine  the  success  of  the  training  program.  Classification  of  the  studies  is 
shown  in  Table  16.  The  information  presented  below  has  been  derived  from  Cook  and  Campbell 
(1979)  and  Goldstein  (1993).  Interested  readers  should  consult  these  two  sources  for  more  detailed 
information  on  quasi-experimental  designs. 
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Table  16. 


Approximate  Design  Used  by  Researchers  in  the  Reviewed  Studies 


Quasi -experimental  design  used _ Researcher(s) 


^  -  -  - - ® - 

X  0 

one-group  posttest-only 

Binket  al.,  1995 

Biner  et  al.,  1994 
Gunawardena  &  Zittle,  1997 

XaO 

Xb  0 

two-group  posttest-only 

Wetzel  et  al.,  1996b 

Souder,  1993 

X  0 

0 

posttest-only  with 
nonequivalent  comparison  group 

Stone,  1988 

Xa  0 

Xb  0 

0 

two-group  posttest-only  with 
nonequivalent  comparison  group 

Wetzel,  1996 

Wetzel  et  al.,  1996a 

X  0,  O2 

one-group  multiple  posttest 

Fellers  &  Moon,  1994 
Oxford  et  al.,  1993 

Oi  X  O2 

one-group  pretest-posttest 

Bramble  &  Martin,  1995 
Coogle  et  al.,  1996 

Wisher  &  Cumow,  1998 

Oi  X  O2 

Oi  O2 

two-group  pretest-posttest  with 
nonequivalent  comparison  group 

Lennon  &  Payne,  1997 
Keene  &  Cary,  1990 

Larson  &  Pruning,  1996 
Phelps  et  al.,  1991 

Oi  X  O2  O3 

Oi  O2  O3 

two-group  pretest-posttest  with 

Wisher  et  al.,  1997 

Wisher  &  Priest,  1998 
Simpson  et  al.,  1995 

nonequivalent  comparison  group 
(multiple  posttests) 


X  =  presence  of  training  delivered  via  DL 
O  =  administration  of  performance  or  attitude  measure 
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To  say  that  a  study  of  a  DL  training  program  possess  internal  validity  means  that  it  can 
be  established  that  the  cause  or  treatment  (e.g.,  interactivity  of  DL  instruction)  was  responsible 
for  the  outcomes  or  effects  of  the  program  (e.g.,  performance  or  satisfaction).  If  internal  validity 
can  not  be  demonstrated  due  to  the  research  design  or  to  the  inability  to  eliminate  other  possible 
causes,  then  the  researcher  may  not  conclude  that  the  treatment  “worked”  (i.e.,  caused  the  higher 
performance).  Some  researchers  use  designs  in  their  studies  that  fail  to  eliminate  many 
alternative  reasons  for  the  consequences  of  the  training  program.  This  usually  occurs  because  of 
the  lack  of  a  comparison  group  and/or  failure  to  obtain  more  than  one  measure  of  performance. 
Designs  which  use  equivalent  comparison  groups  or  include  pretest  and  posttest  measures  can 
make  the  results  of  DL  studies  more  meaningful.  Below  are  just  a  few  of  the  “threats  to  internal 
validity”  or  alternative  explanations  for  the  results  that  can  occur  in  studies  of  DL  training.  In 
each  case,  the  researcher  may  mistakenly  attribute  success  or  failure  to  the  DL  instruction  when 
it  may  have  been  due  to  another  cause: 

history  -  Changes  in  performance  or  attitude  may  be  due  to  another  specific  event,  other 
than  the  treatment  or  use  of  DL.  For  example,  students  may  have  learned  the  material  from  a 
source  outside  of  class  or  were  inspired  to  seek  out  other  information  outside  of  class. 

maturation  -  Changes  in  performance  or  attitude  measures  may  be  due  to  students 
becoming  less  interested  in  the  program  or  more  fatigued  over  time. 

testing  -  Pretest  measures  may  sensitize  students  to  the  knowledge-based  items  and  they 
may  score  higher  on  the  posttest  regardless  of  the  content  of  the  training  program. 

instrumentation  -  The  subjective  test  scoring  may  change  between  test  administrations. 
Also,  the  second  administration  of  an  objective  test  may  be  easier  or  more  difficult  than  the  first. 

mortality  -  Students  with  less  ability,  motivation,  or  time  resources  may  become 
discouraged  and  drop  out  during  the  program  so  that  the  average  posttest  knowledge-based  scores 
are  higher  than  the  average  pretest  scores. 

dijfusion  of  treatments  -  Students  in  local  and  remote  DL  programs  or  students  in  the  DL 
group  and  comparison  groups  may  discuss  information  or  share  experiences  thereby  reducing 
differences  between  the  two  groups.  Also,  students  in  the  “traditional”  comparison  group  may 
perform  differently  when  they  learn  of  the  additional  efforts  and  attention  paid  to  students  in  the 
more  “desirable”  DL  group. 
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Examples 

Several  of  the  studies  (e.g.,  Bink  et  al,  1995;  Gunawardena  &  Zittle,  1997)  used  a  one- 
group  posttest-only  design,  whereby  students  were  given  a  test  of  knowledge  following  the 
administration  of  the  DL-based  course  (see  Table  16).  This  design  is  generally  uninterpretable  due 
to  the  lack  of  a  pretest  measure  of  knowledge  or  a  comparable  comparison  group  of  students  who 
were  not  administered  the  course  via  DL  media.  For  example,  if  students,  on  average,  scored  90% 
on  a  knowledge-based  test,  it  would  be  impossible  to  determine  using  this  information  whether  the 
program  was  successful  or  not.  A  pretest  may  have  revealed  that  students  already  knew  90%  of 
the  material  prior  to  any  instruction,  or  a  comparison  group  of  students  may  have  scored  98%,  on 
average,  after  given  the  instruction  in  a  traditional  classroom. 

The  use  of  pretests  and  comparison  groups  do  not  always  reduce  threats  to  internal  validity. 
Often  the  comparison  group  is  dissimilar  to  the  DL  group.  DL  students  appear  to  be  more 
successful,  in  part,  because  they  tend  to  be  older,  more  motivated  and  self-disciplined,  are  more 
likely  to  possess  a  college  degree,  and  have  expectations  for  higher  grades  (Gottschalk,  1996). 
Comparing  the  posttest  performance  measures  of  these  students  with  those  of  typical  18  to  22-year 
old  college  students  who  may  be  taking  classwork  in  a  traditional  classroom  may  lead  to 
meaningless  or  incorrect  conclusions.  In  addition,  of  the  studies  reviewed  here,  pretest  and  posttest 
measures  of  performance  were  nearly  always  given  only  one  time  (see  Fellers  &  Moon,  1994; 
Simpson  et  al.,  1995;  and  Wisher  et  al,  1997  for  exceptions).  Stronger  conclusions  may  be  drawn 
from  designs  where  multiple  measures  are  included  to  identify  where  learning  occurs  during  the 
training  program.  This  is  important  from  a  cost-savings  perspective  as  well.  Multiple  measures 
given  during  a  lengthy  training  program  may  reveal  that  DL  media  is  most  efficiently  used  during 
specific  periods  of  training  where  the  majority  of  learning  occurs  and  that  low  cost  alternatives, 
such  as  print  or  video  media,  may  be  used  during  other  periods  of  the  training. 

Suggestions  for  Design 

The  best  solution,  of  course,  is  to  randomly  assign  students  from  the  same  population  to 
different  DL  and  comparison  groups.  This  will  eliminate  most  threats  to  internal  validity  and,  if 
reliable  and  valid  measures  are  used,  can  contribute  to  interpretable  findings.  Random 
assignment,  however,  is  usually  impossible,  or  at  least  impractical,  in  many  military  and 
industrial  settings.  It  is  possible,  albeit  difficult,  in  some  academic  settings.  A  reasonable 
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alternative  was  used  by  Miller,  McKenna,  and  Ramsey  (1993)  whereby  instructors  alternated 
locations  for  the  origination  site  of  the  broadcast  so  that  all  students  participated  in  both  the  DL 
and  comparison  conditions  during  two  consecutive  classes.  In  this  manner,  each  student  served 
as  his  or  her  own  comparison  group.  The  average  score  on  mastery  tests  of  content  delivered 
“live”  was  significantly  higher  (15%)  than  those  delivered  via  two-way  video/two-way  audio 
media.  It  is  worth  noting  that  the  Miller  et  al.  study  used  a  stronger,  more  interpretable  design 
than  any  of  the  studies  reviewed  here,  yet  was  the  lone  exception  in  the  57  studies  reviewed  by 
Howard  (1997)  where  performance  was  lower  in  DL  than  in  traditional  classrooms. 

In  most  cases  where  random  assignment  is  not  practical,  there  are  several  ways  for 
researchers  to  plan  and  construct  an  interpretable  design: 

1 .  Administer  a  reliable  (i.e.,  consistent)  and  valid  (i.e.,  accurate  and  free  of  bias)  pretest 
on  the  performance  and  attitudinal  measures  of  interest. 

2.  Administer  these  measures  to  a  comparable  comparison  group. 

3.  If  no  comparable  comparison  group  is  available,  seek  information  about  the  students 
in  terms  of  demographics,  experience,  background,  and  ability  for  use  as  a  covariate. 

4.  Use  multiple  performance  measures  and  administer  the  same  measures  on  several 
occasions;  be  sure  to  use  the  correct  data  analysis  procedures. 

5.  Use  multiple  pretests,  especially  when  measuring  attitudes  that  fluctuate  over  time. 

6.  Plan  ahead  to  determine  which  threats  to  internal  validity  can  be  reduced  through 
design  and  which  threats  cannot  be  ruled  out. 

Related  Issues 

We  have  concentrated  on  specific  areas  of  the  research  literature,  a  representative  sample 
of  empirical  studies  with  adequate  research  qualities  oriented  to  training.  We  will  now  consider 
some  issues  and  findings  common  to  training  and  education:  interaction  between  student  and 
instructor,  individual  differences,  and  the  cost-effectiveness  of  the  use  of  DL  media. 

Interactions 

Although  a  common  belief  is  that  “teachers  teach  the  way  they  were  taught”,  it  may  be 
more  common  that  “teachers  teach  the  way  they  learned”  (Dunn  &  Dunn,  1979).  This  maxim 
may  usually  hold  in  the  “traditional”  classroom,  since  nearly  all  instructors  have  been  educated 
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primarily  in  the  traditional  classroom.  In  a  specific  DL  format,  however,  the  available 
technology  may  exclude  the  methods  by  which  instructors  have  refined  over  their  teaching 
careers.  For  example,  if  certain  instructors  rely  on  the  use  of  small  group  discussions  or  team 
projects  in  a  course,  they  may  find  themselves  stymied  or  their  strong  teaching  skills  neutralized 
if  they  teach  the  course  via  audio  teletraining  or  one-way  video  /  two-way  audio  media. 

Mosston  and  Ashworth  (1990)  argued  that  teaching  and  all  the  in-class  behaviors  of 
instruction  are  governed  by  decision  making,  and  that  these  decisions  occur  before,  during,  and 
after  the  actual  classroom  instruction.  A  good  instructor,  therefore,  is  one  who  makes  good 
decisions.  We  would  add  that  good  decision-makers  also  need  to  be  good  information-seekers. 
This  information  seeking  may  occur  before  (e.g.,  choice  of  instruction  materials),  during  (e.g., 
reading  the  puzzled  looks  of  students),  or  after  (e.g.,  obtaining  useful  and  valid  feedback  from 
evaluation  instruments)  the  classroom  instruction.  With  the  exception  of  Web-based  DL 
technology,  however,  the  amount  and  scope  of  information  seeking  may  be  severely  reduced. 
This  reduction  of  information  may  lead  to  poorer  decision  making  and,  in  turn,  poorer 
instruction.  One  way  to  increase  this  information  is  through  increased  interactions  during 
learning. 

Wagner  (1994)  defines  interactions  as  reciprocal  events  requiring  two  objects  and  two 
actions.  Such  interactions  foster  behaviors  in  which  individuals  and  groups  influence  one 
another.  Interactivity,  on  the  other  hand,  is  considered  by  Wagner  to  be  the  electronic  connection 
between  the  origination  site  and  the  remote  sites,  a  technological  connection  rather  than  a 
behavioral  exchange.  Interactivity  focuses  on  the  properties  of  the  DL  configuration  such  as  a 
push-to-talk  microphone  or  an  e-mail  capability  in  computer-mediated  conferencing. 

Wagner  (1994)  identifies  13  types  of  interactions  that  can  occur  in  DL.  The  hallmark  of 
interactions  is  that  they  must  result  in  the  transfer  of  knowledge  or  a  change  in  intrinsic 
motivation.  The  types  of  interactions  are  listed  in  Table  17. 
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Table  17. 

Interactions  That  Can  Occur  in  DL 


Types  of  Interactions 

to  increase  willingness  to  engage  in  learning 

to  increase  participation 

to  develop  communication 

to  receive  feedback 

to  enhance  elaboration  and  retention 

to  support  leamer/self-regulation 

to  increase  motivation 


for  negotiation  of  understanding 
for  teambuilding 
for  discovery 
for  exploration 

for  clarification  of  understanding 
for  closure 


In  our  review  of  the  training  literature,  the  categories  of  interactions  were  never  specified 
in  the  reports.  Based  on  our  reading,  it  seems  safe  for  us  to  assume  that  all  of  these  interactions 
were  applied  during  the  course  of  training  over  5,400  students  in  a  wide  variety  of  topics  using 
various  distance  learning  technologies.  For  future  work,  it  would  be  advantageous  to  begin 
studying  the  relative  effect  on  learning  of  the  interactions  as  applied  to  different  types  of 
students,  tasks,  and  training  technologies. 

Individual  Differences 

Although  much  has  been  written  about  the  technology  that  is  used  to  teach  over  a 
distance,  less  has  been  written  about  the  students  who  are  attempting  to  learn  using  this 
technology.  It  is  unlikely  that  all  students,  regardless  of  background,  experience,  maturity,  and 
other  individual  characteristics,  learn  equally  well  through  distributed  technologies.  A  crucial 
issue,  therefore,  is  to  identify  which  individual  learning  characteristics  differentiate  between  those 
who  learn  better  than  others  via  technology. 

Several  such  individual  difference  measures  have  been  identified  in  the  literature. 

Among  those  are  three  broad  categories  that  are  often  used  interchangeably:  learning  style, 
cognitive  style,  and  learning  strategy.  There  is  sharp  disagreement  among  researchers 
concerning  which  category  is  broadest  and  how  each  are  defined  (Curry,  1990;  Shih,  Ingebritsen, 
Pleasants,  Flickinger,  &  Brown,  1998).  However,  since  the  term  “learning  style”  appears  most 
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frequently  in  the  literature  and  is  represented  by  the  largest  number  of  existing  measures,  we  will 
reserve  our  discussion  to  this  topic. 

Learning  style.  This  term  can  be  generally  defined  as  students’  existing  learning 
strengths  or  preferred  manner  of  using  their  intellectual  abilities  (Sternberg  &  Grigorenko,  1997). 
It  is  presumed  that  learning  styles  are  stable  and  relatively  permanent  characteristics  of  an 
individual  (Garger  &  Guild,  1984).  Although  the  concept  has  a  long  history  in  psychology  and 
among  educators,  it  has  broadened  its  appeal  with  the  advent  of  technology-mediated  training 
and  educational  tools  in  academic,  industry,  and  military  settings.  It  is  now  recognized  that 
distance  learning  materials  must  facilitate  the  desired  learning  outcomes  in  a  way  that  is 
compatible  with  students’  learning  styles  (Council  of  Chief  State  School  Officers,  1995).  The 
choice  of  learning  style  measure  to  use,  however,  is  difficult.  A  recent  review  (Champagne, 
Dooner,  &  Tunney,  1998)  uncovered  64  measures  of  learning  style  in  print  as  well  as  several 
dozen  learning  styles  available  for  use  on  the  Internet.  The  quality  and  validity  of  many  of  these 
measures  have  been  called  into  question  (Curry,  1990),  particularly  those  available  on  the 
Internet.  For  a  review  of  various  learning  style  measures  see  Champagne  et  al.  (1998),  Curry 
(1987),  and  DeBello  (1990). 

Research  suggests  that  students  whose  learning  styles  are  compatible  with  the  style  of 
teaching  delivered  in  the  classroom  may  perform  higher  than  students  whose  learning  styles  are 
not  compatible  with  such  an  instructional  style  (Fumham,  1992;  Honey  &  Mumford,  1986; 
Ingham,  1991).  Oxford,  Park-Oh,  Ito,  and  Sumrall  (1993)  classified  learners  of  a  satellite 
program  to  learn  Japanese  into  the  sensory  preferences  of  visual,  auditory,  or  haptic  modalities. 
Although  the  researchers  did  not  directly  test  performance  differences,  they  found  that  students 
with  an  auditory  preference  were  more  motivated  than  visual  students,  and  both  were  more 
motivated  than  the  students  with  a  haptic  preference. 

Douzenis  (1998)  administered  both  the  Group  Embedded  Figures  Test  (Witkin,  Oltman, 
Raskin,  &  Karp,  1971)  and  the  Learning  Style  Inventory  (Kolb,  1976)  to  masters-level  graduate 
students  in  a  DL  course  in  educational  research.  She  found  that  the  Group  Embedded  Figures 
Test  and  one  of  the  four  factors  of  the  Learning  Style  Inventory  (accommodator)  were  significant 
predictors  of  achievement.  We  agree  with  the  author  that  although  the  amount  of  total  variance 
explained  was  low,  the  use  of  multiple  learning  style  measures  is  a  worthwhile  approach.  In 
contrast  to  Douzenis,  a  study  by  Shih  et  al.  (1998)  found  the  Group  Embedded  Figures  Test  to  be 
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an  unsatisfactory  predictor  of  performance.  In  Web-based  courses  in  Zoology  and  Biology,  they 
found  no  significant  differences  on  a  standardized  achievement  test  between  students  categorized 
as  field  dependent  or  field  independent. 

Self-efficacy.  In  addition  to  learning  style,  there  are  many  other  individual  difference 
measures  in  print  that  have  shown  acceptable  psychometric  properties  and  which  evaluators  of 
DL  programs  may  wish  to  administer.  Self-efficacy  is  the  confidence  one  has  in  being  able  to 
succeed  at  a  particular  task  due  to  beliefs  of  one’s  skills,  knowledge,  and  abilities  (Bandura,  1982). 
Gist  (1989)  developed  a  popular  paper-and-pencil  measure  of  this  concept  and  Maurer  and  Pierce 
(1998)  recently  developed  a  similar,  more  practical,  version  of  the  measure.  High  self-efficacy  has 
been  demonstrated  to  predict  higher  performance  on  physical  exercise  tasks  (Bandura  &  Cervone, 
1986),  higher  levels  of  mastery  on  a  software  training  program  (Gist,  Schwoerer,  &  Rosen,  1989), 
and  higher  productivity  among  university  faculty  (Taylor,  Locke,  Lee,  &  Gist,  1984). 

Motivation.  Although  student  motivation  would  intuitively  appear  to  be  an  important 
individual  difference  variable,  it  is  a  difficult  construct  to  operationalize.  By  definition, 
motivation  is  that  which  energizes,  directs,  and  sustains  behavior  (Steers  &  Porter,  1991). 
However,  there  is  no  single  agreed-upon  measure  for  motivation,  and  there  exists  many 
competing  and  complimentary  theories  containing  various  components,  sources  of  motivation, 
and  predicted  outcomes.  Oxford  et  al.  (1993)  found  motivation  to  be  the  most  important 
predictor  of  success  in  learning  a  new  language  in  DL  when  using  specific  measure  of  student 
motivation. 

To  summarize  the  research  in  this  area,  the  question  is  not  whether  learning  style,  in 
general,  is  good  or  poor  predictor  of  student  attitudes,  learning,  and  performance.  Researchers 
should,  instead,  systematically  determine  which  of  the  nearly  70  existing  learning  style  measures 
are  the  best  predictors  of  certain  criteria,  and  which  need  further  validation  before  they  may  be 
found  useful. 


COST  EFFECTIVENESS  OF  DISTANCE  LEARNING 
As  described  earlier,  cost-effectiveness  is  viewed  differently  by  industry,  the  government, 
and  the  military  when  compared  to  academia.  The  former  seek  to  save  travel,  per  diem,  and 
productivity  costs  by  keeping  employees  near  the  workplace  rather  than  at  a  distant  training  site. 
Here,  costs  for  procuring  a  DL  facility  and  developing  or  purchasing  courses  can  be  justified 
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with  training  quality  that  is  “as  good  as”  the  training  received  at  a  distant  site.  Examples  of  this 
were  reported  in  Wisher  and  Priest  (1998)  in  which  audio  teletraining  of  unit  clerks  enabled  the 
Army  National  Guard  to  avoid  costs  exceeding  $1,000  per  trainee  for  a  three-week  course. 

When  considering  that  the  yearly  full  time  equivalent  training  load  in  the  Defense  Department  is 
165,000  “students”,  making  converting  what  is  appropriate  to  convert  to  a  DL  format  can  reap 
tremendous  monetary  benefits.  In  another  study  we  reviewed,  Phelps  et  al.  (1991)  determined 
that  converting  selective  modules  of  certain  courses  for  Army  officer  training  become  cost- 
effective  after  ten  iterations.  This  study  was  conducted  years  ago  when  the  costs  to  supply  each 
student  with  a  personal  computer  were  much  higher  than  it  is  today.  In  industry,  calls  to  “trim 
travel  budgets  with  distance  learning”  (Miller,  1991)  are  becoming  more  pronounced  in  efforts  to 
maintain  a  skilled  workforce  at  reasonable  costs. 

In  academia,  the  majority  of  students  already  reside  at  or  near  the  traditional  college 
campus.  Efforts  are  made  to  convert  courses  to  a  DL  format  to  attract  students  who  otherwise 
might  not  attend  that  institution  because  of  travel  or  time  constraints.  The  educational 
institutions  do  not  have  to  pay  travel  or  per  diem  costs,  so  being  “as  good  as”  the  traditional 
classroom  is  not  necessarily  a  sufficient  justification  for  course  conversion.  Rather,  training 
must  add  educational  value  and  be  flexible  and  of  high  quality  in  order  to  be  “better  than”  the 
programs  of  other  educational  institutes. 

The  competition  among  educational  providers  is  changing  the  economics  of  higher 
education  as  the  supply-side  of  DL  challenges  the  industrial-age  traditions  of  education.  There  is 
a  recent  recognition  of  the  importance  of  attracting  not  only  students  from  within  the  worldwide 
market  of  learners,  but  also  attracting  groups  of  individuals  who  find  travel  to  campuses  to  be 
difficult  or  inconvenient.  These  groups  include  those  who  are  homebound  because  of  child  care 
responsibilities  or  a  disability,  full-time  employees  involved  in  life-long  learning  programs  or 
retraining  for  a  career  change,  and  in-service  training  for  teachers  and  educators.  In  addition, 
universities  have  discovered  the  financial  advantages  of  delivering  DL-based  professional 
masters  programs  to  a  new  market  of  students.  Unfortunately,  it  is  difficult  to  assess  the  costs  of 
DL  training  courses  in  academia  due  to  the  variability  of  technologies  used  as  well  as  the  myriad 
relevant  costs  (e.g.,  marketing,  faculty,  teaching  assistant,  and  production  manager  salaries, 
technical  support).  It  is  not  unusual  for  a  single  training  program  to  be  simultaneously  delivered 
via  three  or  more  technologies  (e.g.,  Web-based,  VTT,  videotape)  or  in  both  synchronous  and 
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asynchronous  fashion,  depending  on  the  needs  or  resources  of  the  students.  Therefore,  the 
complex  question  of  the  cost-effectiveness  of  DL  technology  may  have  to  be  answered  on  an 
individual  course  or  program  basis. 


FUTURE  WORK 
Learning  and  Training  Outcomes 

One  recommendation  that  arises  from  our  review  of  the  research  involves  the  careful, 
systematic  use  of  learning  and  training  outcomes  and  the  measures  that  represent  those 
outcomes.  Evaluation  of  training  programs  is  primarily  conducted  to  determine  if  training 
objectives  were  achieved  (Campbell,  1988)  as  well  as  whether  accomplishing  those  objectives 
enhanced  performance  on  the  job  (Kraiger,  Ford,  &  Salas,  1993).  The  most  popular  model  used 
to  evaluate  training  programs  has  been  that  proposed  by  Kirkpatrick  (1994)  which  consists  of 
trainee  reactions,  learning,  behavior,  and  organizational  results.  Also,  Kraiger  et  al.  (1993)  have 
developed  a  multidimensional  perspective  to  learning  outcomes  that  is  firmly  couched  in  existing 
theories  and  taxonomies  (e.g..  Bloom,  1965;  Gagne,  1984).  Their  classification  integrates  three 
learning  outcomes  (cognitive,  skill-based,  affective),  with  their  representative  learning  constructs 
(e.g.,  verbal  knowledge,  attitudes,  motivation),  and  suggests  particular  methods  to  successfully 
measure  these  constructs  (e.g.,  self-report,  protocol  analysis,  recognition  and  recall  tests).  This 
framework  of  finding  appropriate  measures  for  the  appropriate  outcomes  that  represent  the 
critical  constructs  is  one  example  of  the  careful  consideration  of  theory  and  research  when 
designing  evaluations  of  DL  programs. 


Evaluation 

The  evaluation  of  many  DL  programs  currently  delivered  by  academic,  industrial,  and 
military  institutions  are  much  like  autopsies.  Once  the  class  is  over,  they  attempt  to  discern  what 
went  wrong.  Although  educators  have  the  technology  to  deliver  courses  to  nearly  every  location 
in  the  world,  they  usually  rely  on  end-of-training,  paper-and-pencil  measures  of  student  attitudes 
or  performance  that  often  are  of  questionable  validity  and  reliability.  These  measures  are  given 
too  infrequently  to  make  immediate  improvements  in  the  delivery  of  training.  What  is  needed 
are  reliable  and  valid  measures  of  performance  which  can  be  immediately  administered  to  the 
students  and  quickly  returned  to  the  instructor,  while  the  class  is  still  in  session.  This  would 
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provide  quick  feedback  to  the  students,  demonstrate  to  students  that  they  are  an  important  part  of 
the  training  process,  and  help  to  quickly  improve  the  delivery  of  training  with  minimal  time 
expenditure  from  the  instructor  (Champagne,  1998). 

Successful  DL  programs  will  continue  to  require  a  stronger  and  more 
comprehensive  evaluation  component.  Evaluation  is  unlikely  to  be  successful  if  performed 
as  an  “add  on”  or  conducted  by  someone  without  the  proper  skills  in  evaluation  theory  and 
methodology,  experimental  and  quasi-experimental  design,  criteria  development, 
measurement  theory,  and  statistical  analysis 

T  The  Future  ofDL  in  Academia  and  Industry 

With  the  increasing  reliance  on  technology,  there  is  little  doubt  that  the  Internet  (and  its 
future  offspring)  will  continue  to  be  an  important  resource  in  education  and  training.  Many  have 
suggested  that  the  use  of  this  technology  will  soon  allow  private  industry  to  challenge 
universities  for  the  delivery  of  education.  Indeed,  online  law  schools  already  exist  which  feature 
prestigious  faculty  from  various  universities.  Students  and  professionals  will  soon  be  no  longer 
constrained  by  time  or  place  for  education.  Within  a  short  time,  nearly  every  major  learning 
institution  and  many  private  organizations  will  be  able  to  educate  and  train  people  at  a  distance, 
and  students  from  across  the  globe  will  select  where  to  send  their  tuition  dollars  based  on  factors 
other  than  location.  Training  and  education  programs  that  are  long  on  technological  gimmickry 
but  short  on  foundational  research  as  to  why  that  technology  would  be  effective  for  successful 
learning  will  not  sustain  their  programs  in  a  competitive  market. 

The  decreasing  costs  and  increasing  popularity  of  information  technology  has 
dramatically  changed  the  delivery  of  training  in  the  past  decade.  However,  the  technology  has 
quickly  outpaced  the  theory  that  supports  its  effectiveness,  and  the  application  of  technology  has 
surpassed  the  evaluation  of  that  technology.  There  is  no  body  of  research  that  meaningfully 
unites  training  objectives,  training  content,  instructional  style;  and  the  DL  media.  What  is 
needed  is  empirical  information  and  systematic  research  which  would  help  instructors  design 
their  courses  to  incorporate  the  most  appropriate  technology  based  on  the  course  content,  the 
type  of  student  attending  the  course,  and  the  type  of  teaching  style  used  by  the  instructor. 
Crawford  and  Suchan  (1996)  drew  a  similar  conclusion  in  their  call  for  systematic  selection  of 
“instructional  media  for  specific  learning  applications  that  places  priority  on  the  desired  learning 
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outcome  and  the  media  required  to  support  the  instructional  techniques  to  attain  that  outcome 
(p.  36). 


SUMMARY  REMARKS 

The  studies  reviewed  in  this  report  are  organized  in  Table  16  in  order  of  least 
interpretable  designs  to  most  interpretable  designs.  The  nine  studies  that  used  only  posttest 
measures  of  performance  or  student  attitudes,  some  with  a  nonequivalent  comparison  group,  are 
difficult  to  interpret.  Without  a  pretest  measure,  many  threats  (e.g.,  maturation,  history, 
mortality)  to  the  internal  validity  of  the  study  cannot  be  ruled  out.  That  is,  the  scores  on  the 
posttest  may  be  due  to  other  factors  besides  the  DL  media.  Stronger  designs  are  those  which  use 
multiple  posttests,  which  may  help  to  rule  out  instrumentation  or  maturation  effects  as  an 
alternative  explanation,  and  those  designs  using  a  comparison  group,  which  may  help  to  rule  out 
mortality  and  history  effects. 

Several  conclusions  may  be  drawn  based  on  the  designs  of  these  studies.  First,  because 
random  assignment  of  students  to  the  DL  and  comparison  groups  was  not  possible,  none  of  the 
studies,  individually,  can  reject  the  possibility  that  the  positive  results  were  due  to  other  factors 
besides  the  use  of  DL  media.  Some  could  argue,  however,  that  the  uniform  positive  results, 
aggregated  across  all  studies,  would  suggest  that  the  superiority  or  equivalence  of  the  DL  groups 
is  a  meaningful  finding.  Future  studies  which  allow  random  assignment  of  students  to  groups  or 
that  measure  performance  while  students  alternate  between  groups  (e.g..  Miller  et  al.,  1993) 
would  allow  more  conclusive  results  to  be  drawn. 

Second,  the  use  of  “convenience  samples”  to  serve  as  comparison  or  DL  groups  makes  it 
difficult  to  draw  meaningful  conclusions.  In  most  cases,  there  are  specific  reasons  (e.g.,  location, 
availability  of  technology,  convenience,  and  job  requirements)  why  students  attend  training 
programs  via  DL  media.  There  may  be  fundamental  pre-existing  differences  between  students 
who  choose  to  attend  the  training  program  or  choose  a  DL-based  training  program  over  a  non- 
DL  based  training  program  that  are  reflected  in  the  measures  of  performance. 

Third,  nearly  all  the  studies  reviewed  suggested  that  differences  between  groups  was 
solely  due  to  the  DL  media  used  rather  than  to  individual  student  differences  such  as  learning 
style,  self-efficacy,  or  motivation.  In  a  traditional  classroom,  most  instructors  realize  that  not  all 
students  learn  best  using  the  same  method.  Some  students  prefer  a  hands-on  approach;  some 
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prefer  to  work  with  teammates;  others  gain  knowledge  by  just  reading  the  text  themselves;  and 
still  others  find  that  real-life  examples  or  stories  from  their  instructors  are  most  informative. 

This  recognition  of  individual  differences  in  the  traditional  classroom  should  extend  to  the  DL 
classroom.  Future  studies  should  incorporate  measures  of  individual  differences  to  avoid 
ambiguous  or  erroneous  evaluation  results  that  assume  all  students  in  a  training  program  possess 
the  same  skills,  knowledge,  and  abilities. 

This  report  examined  the  research  methodologies,  experimental  designs,  variables, 
comparison  groups,  sample  sizes,  and  other  factors  that  lead  many  to  conclude  that  distance 
learning  is  an  effective  alternative  to  the  traditional  training  classroom.  Based  on  our  review  of 
the  literature,  it  is  difficult  to  identify  with  precision  why  DL  appears  to  be  successful:  was  it  the 
effort  in  course  design?  the  interactions  made  possible  by  method  of  communication?  the 
instructional  technique?  Details  in  the  literature  are  usually  incomplete.  The  interpretability  of 
the  training  studies  varies.  Unfortunately,  researchers  are  often  bound  by  the  realities  of  their 
test  environments.  With  improved  designs  and  stronger  controls,  the  research  community  should 
better  be  able  to  inform  policy  makers,  training  managers,  and  practitioners  where  best  to  invest 
in  distance  learning. 

As  the  Army  pursues  it  goal  to  convert  over  500  courses  to  a  DL  format  and  install 
hundreds  of  DL  classrooms  around  the  world,  efforts  should  be  taken  for  stronger  evaluations  of 
the  effectiveness  of  DL.  In  particular,  stronger  experimental  designs  should  be  followed,  when 
practical,  to  eliminate  alternate  explanations  as  to  why  a  DL  course  was  effective.  Such  stronger 
designs  will  allow  a  better  understanding  of  the  relative  return  on  a  training  investment  —  from 
the  course  design,  the  interactions  made  possible  by  the  training  technology,  or  the  instructional 
techniques  inherent  in  a  particular  distance  learning  technology. 
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